You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "danny0405 (via GitHub)" <gi...@apache.org> on 2023/07/16 03:47:39 UTC

[GitHub] [hudi] danny0405 opened a new pull request, #9209: [WIP][HUDI-6539] New LSM tree style archived timeline

danny0405 opened a new pull request, #9209:
URL: https://github.com/apache/hudi/pull/9209

   ### Change Logs
   
   A new LSM style archived timeline.
   
   ### Impact
   
   none
   
   ### Risk level (write none, low medium or high below)
   
   none
   
   ### Documentation Update
   
   _Describe any necessary documentation update if there is any new feature, config, or user-facing change_
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1661883911

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 6ab7d4bd2696988e3784291879d1f713d6a9c98d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978) 
   * 4ade37c10c908c0422915aaa489208e6ee62bb0d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280073859


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveInstant.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * A combination of instants covering action states: requested, inflight, completed.
+ */
+public class ActiveInstant implements Serializable, Comparable<ActiveInstant> {
+  private final HoodieInstant requested;
+  private final HoodieInstant inflight;
+  private final HoodieInstant completed;
+
+  /**
+   * The constructor.
+   */
+  protected ActiveInstant(@Nullable HoodieInstant requested, @Nullable HoodieInstant inflight, HoodieInstant completed) {
+    this.requested = requested;
+    this.inflight = inflight;
+    this.completed = completed;
+  }
+
+  public static ActiveInstant fromInstants(List<HoodieInstant> instants) {
+    ValidationUtils.checkArgument(instants.size() <= 3);
+    HoodieInstant requested = null;
+    HoodieInstant inflight = null;
+    HoodieInstant completed = null;
+    for (HoodieInstant instant : instants) {
+      if (instant.isRequested()) {
+        requested = instant;
+      } else if (instant.isInflight()) {
+        inflight = instant;
+      } else {
+        completed = instant;
+      }
+    }
+    return new ActiveInstant(requested, inflight, Objects.requireNonNull(completed));
+  }
+
+  public List<HoodieInstant> getPendingInstants() {
+    List<HoodieInstant> instants = new ArrayList<>(2);
+    if (this.requested != null) {
+      instants.add(this.requested);
+    }
+    if (this.inflight != null) {
+      instants.add(this.inflight);
+    }
+    return instants;
+  }
+
+  public HoodieInstant getCompleted() {
+    return completed;
+  }
+
+  public String getAction() {
+    return this.completed.getAction();
+  }
+
+  /**
+   * A COMPACTION action eventually becomes COMMIT when completed.

Review Comment:
   +1 we can decouple this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280269996


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.

Review Comment:
   https://issues.apache.org/jira/browse/HUDI-6626



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1647423384

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1278794293


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>

Review Comment:
   No, the cleaning of the LSM tree legacy files itself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1278790424


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,

Review Comment:
   Yeah, size based compaction makes sense to me, we can always pick the oldest files from a layer but also control the gross file size of source files to be under 1GB.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1647786249

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799) 
   * 62001bbc6c5d9306f95d4bca3dd2bada3ca5c898 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288095424


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  private final Map<String, byte[]> readCommits;
+
+  public LegacyArchivedMetaEntryReader(HoodieTableMetaClient metaClient) {
+    this.metaClient = metaClient;
+    this.readCommits = new HashMap<>();
+  }
+
+  /**
+   * @deprecated Use {@code HoodieArchivedTimeline#readCommit} instead.
+   */
+  private HoodieInstant readCommit(GenericRecord record, boolean loadDetails) {
+    final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
+    final String action = record.get(ACTION_TYPE_KEY).toString();
+    final String stateTransitionTime = (String) record.get(STATE_TRANSITION_TIME);
+    if (loadDetails) {
+      getMetadataKey(action).map(key -> {
+        Object actionData = record.get(key);
+        if (actionData != null) {
+          if (action.equals(HoodieTimeline.COMPACTION_ACTION)) {
+            this.readCommits.put(instantTime, HoodieAvroUtils.indexedRecordToBytes((IndexedRecord) actionData));
+          } else {
+            this.readCommits.put(instantTime, actionData.toString().getBytes(StandardCharsets.UTF_8));
+          }
+        }
+        return null;
+      });
+    }
+    return new HoodieInstant(HoodieInstant.State.valueOf(record.get(ACTION_STATE).toString()), action,
+        instantTime, stateTransitionTime);
+  }
+
+  @Nonnull
+  private Option<String> getMetadataKey(String action) {
+    switch (action) {
+      case HoodieTimeline.CLEAN_ACTION:
+        return Option.of("hoodieCleanMetadata");
+      case HoodieTimeline.COMMIT_ACTION:
+      case HoodieTimeline.DELTA_COMMIT_ACTION:
+        return Option.of("hoodieCommitMetadata");
+      case HoodieTimeline.ROLLBACK_ACTION:
+        return Option.of("hoodieRollbackMetadata");
+      case HoodieTimeline.SAVEPOINT_ACTION:
+        return Option.of("hoodieSavePointMetadata");
+      case HoodieTimeline.COMPACTION_ACTION:
+      case HoodieTimeline.LOG_COMPACTION_ACTION:
+        return Option.of("hoodieCompactionPlan");
+      case HoodieTimeline.REPLACE_COMMIT_ACTION:
+        return Option.of("hoodieReplaceCommitMetadata");
+      case HoodieTimeline.INDEXING_ACTION:
+        return Option.of("hoodieIndexCommitMetadata");
+      default:
+        LOG.error(String.format("Unknown action in metadata (%s)", action));
+        return Option.empty();
+    }
+  }
+
+  /**
+   * This is method to read selected instants. Do NOT use this directly use one of the helper methods above
+   * If loadInstantDetails is set to true, this would also update 'readCommits' map with commit details
+   * If filter is specified, only the filtered instants are loaded
+   * If commitsFilter is specified, only the filtered records are loaded
+   *
+   * @deprecated Use {@code HoodieArchivedTimeline.loadInstantsV2} instead.
+   */
+  private List<HoodieInstant> loadInstants(
+      HoodieArchivedTimeline.TimeRangeFilter filter,
+      boolean loadInstantDetails,
+      Function<GenericRecord, Boolean> commitsFilter) {
+    try {
+      // List all files
+      FileStatus[] fsStatuses = metaClient.getFs().globStatus(
+          new Path(metaClient.getArchivePath() + "/.commits_.archive*"));
+
+      // Sort files by version suffix in reverse (implies reverse chronological order)
+      Arrays.sort(fsStatuses, new ArchiveLogVersionComparator());
+
+      Set<HoodieInstant> instantsInRange = new HashSet<>();
+      for (FileStatus fs : fsStatuses) {
+        // Read the archived file
+        try (HoodieLogFormat.Reader reader = HoodieLogFormat.newReader(metaClient.getFs(),
+            new HoodieLogFile(fs.getPath()), HoodieArchivedMetaEntry.getClassSchema())) {
+          int instantsInPreviousFile = instantsInRange.size();
+          // Read the avro blocks
+          while (reader.hasNext()) {
+            HoodieLogBlock block = reader.next();
+            if (block instanceof HoodieAvroDataBlock) {
+              HoodieAvroDataBlock avroBlock = (HoodieAvroDataBlock) block;
+              // TODO If we can store additional metadata in datablock, we can skip parsing records
+              // (such as startTime, endTime of records in the block)
+              try (ClosableIterator<HoodieRecord<IndexedRecord>> itr = avroBlock.getRecordIterator(HoodieRecord.HoodieRecordType.AVRO)) {
+                StreamSupport.stream(Spliterators.spliteratorUnknownSize(itr, Spliterator.IMMUTABLE), true)
+                    // Filter blocks in desired time window
+                    .map(r -> (GenericRecord) r.getData())
+                    .filter(commitsFilter::apply)
+                    .map(r -> readCommit(r, loadInstantDetails))
+                    .filter(c -> filter == null || filter.isInRange(c.getTimestamp()))
+                    .forEach(instantsInRange::add);
+              }
+            }
+          }
+
+          if (filter != null) {
+            int instantsInCurrentFile = instantsInRange.size() - instantsInPreviousFile;
+            if (instantsInPreviousFile > 0 && instantsInCurrentFile == 0) {
+              // Note that this is an optimization to skip reading unnecessary archived files
+              // This signals we crossed lower bound of desired time window.
+              break;
+            }
+          }
+        } catch (Exception oriException) {
+          // merge small archive files may leave uncompleted archive file which will cause exception.
+          // need to ignore this kind of exception here.
+          try {
+            if (isCorruptCompactionFile(metaClient, fs)) {
+              continue;
+            }
+            throw oriException;
+          } catch (Exception e) {
+            // If anything wrong during parsing merge archive plan, we need to throw the original exception.
+            // For example corrupted archive file and corrupted plan are both existed.
+            throw oriException;
+          }
+        }
+      }
+
+      ArrayList<HoodieInstant> result = new ArrayList<>(instantsInRange);
+      Collections.sort(result);
+      return result;
+    } catch (IOException e) {
+      throw new HoodieIOException(
+          "Could not load archived commit timeline from path " + metaClient.getArchivePath(), e);
+    }
+  }
+
+  public static boolean isCorruptCompactionFile(HoodieTableMetaClient metaClient, FileStatus fileStatus) throws IOException {
+    Path planPath = new Path(metaClient.getArchivePath(), MERGE_ARCHIVE_PLAN_NAME);
+    HoodieWrapperFileSystem fileSystem = metaClient.getFs();
+    if (fileSystem.exists(planPath)) {
+      HoodieMergeArchiveFilePlan plan = TimelineMetadataUtils.deserializeAvroMetadata(FileIOUtils.readDataFromPath(fileSystem, planPath).get(), HoodieMergeArchiveFilePlan.class);
+      String mergedArchiveFileName = plan.getMergedArchiveFileName();
+      if (!StringUtils.isNullOrEmpty(mergedArchiveFileName) && fileStatus.getPath().getName().equalsIgnoreCase(mergedArchiveFileName)) {
+        LOG.warn("Catch exception because of reading uncompleted merging archive file " + mergedArchiveFileName + ". Ignore it here.");
+        return true;
+      }
+    }
+    return false;
+  }
+
+  /**
+   * Sort files by reverse order of version suffix in file name.
+   */
+  public static class ArchiveLogVersionComparator implements Comparator<FileStatus>, Serializable {

Review Comment:
   It's copied from existing code, should be tested altogether in `TestLegacyArchivedMetaEntryReader`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288119597


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {

Review Comment:
   I don't think we need a LSMTimeline, LSM is just one of the impl, add a LSMTimelineReader and move all the read methods there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288147702


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.

Review Comment:
   > Class LSM ; L0 files may have overlap in ranges, but L1 onward don't. So a single compaction in one level can ripple through to lower levels as well.
   
   I guess you are describing the classic `layered` compaction algorithm, that is to pick up a sub-tree (including several files from current layer and all the overlapping files in next layer), here we choose universal compaciton instead, that is to always compact the constant number of files from the current layer to the next layer and we ignore the overlapping of the  files.
   
   Might need to think through this once we want a global timeline.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1659505428

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 9e7266dffff31156421a129d11dbde82be2fc4fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919) 
   * 1874d4fadea7db5be55f0fd938276eb594258357 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280268703


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>

Review Comment:
   Yes, but is there any possibility the reader see an empty MANIFEST file? Anyway, we now always write a temp file first then rename to the final file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1692939168

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461",
       "triggerID" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 803df61d0d04f7e7403d1177325a365e9bbafab5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226) 
   * 2288afba514c322841e5479a7fc7f6cdc8bf8b2d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6539] New LSM tree style archived timeline [hudi]

Posted by "waywtdcc (via GitHub)" <gi...@apache.org>.
waywtdcc commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1774736557

   Hello, does the master branch now support lsm format merge? @danny0405 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288094060


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java:
##########
@@ -137,6 +141,41 @@ public static HoodieArchivedMetaEntry createMetaWrapper(HoodieInstant hoodieInst
     return archivedMetaWrapper;
   }
 
+  public static HoodieArchivedInstant createArchivedInstant(ActiveInstant activeInstant, HoodieTableMetaClient metaClient) {

Review Comment:
   Already added in `TestMetadataConversionUtils`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288155900


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {

Review Comment:
   First entries based but also limited by the size. We take files number first because this would make the LSM tree file layout more stable (files are near in size with each other in the same layer.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1647495291

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1649331921

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * c281ded6d554350dfe362cce496d6d72cfe0bbbe Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822) 
   * a7f8558aaffdaab4850780224e1385c3e682372a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280262854


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,

Review Comment:
   Finally, we encode the manifet content as a JSON string, which is more friendly to extend in the future if there are new requests. Also, the file size are encoded along with the file name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1661443076

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 9a2afc24ea8552ec6869f9d5819a00c78067a4e6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955) 
   * 6ab7d4bd2696988e3784291879d1f713d6a9c98d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1278642514


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveInstant.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * A combination of instants covering action states: requested, inflight, completed.
+ */
+public class ActiveInstant implements Serializable, Comparable<ActiveInstant> {
+  private final HoodieInstant requested;
+  private final HoodieInstant inflight;
+  private final HoodieInstant completed;
+
+  /**
+   * The constructor.
+   */
+  protected ActiveInstant(@Nullable HoodieInstant requested, @Nullable HoodieInstant inflight, HoodieInstant completed) {
+    this.requested = requested;
+    this.inflight = inflight;
+    this.completed = completed;
+  }
+
+  public static ActiveInstant fromInstants(List<HoodieInstant> instants) {
+    ValidationUtils.checkArgument(instants.size() <= 3);
+    HoodieInstant requested = null;
+    HoodieInstant inflight = null;
+    HoodieInstant completed = null;
+    for (HoodieInstant instant : instants) {
+      if (instant.isRequested()) {
+        requested = instant;
+      } else if (instant.isInflight()) {
+        inflight = instant;
+      } else {
+        completed = instant;
+      }
+    }
+    return new ActiveInstant(requested, inflight, Objects.requireNonNull(completed));
+  }
+
+  public List<HoodieInstant> getPendingInstants() {
+    List<HoodieInstant> instants = new ArrayList<>(2);
+    if (this.requested != null) {
+      instants.add(this.requested);
+    }
+    if (this.inflight != null) {
+      instants.add(this.inflight);
+    }
+    return instants;
+  }
+
+  public HoodieInstant getCompleted() {
+    return completed;
+  }
+
+  public String getAction() {
+    return this.completed.getAction();
+  }
+
+  /**
+   * A COMPACTION action eventually becomes COMMIT when completed.

Review Comment:
   something to think about is - whether we keep it "COMPACTION" with the new changes.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveInstant.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * A combination of instants covering action states: requested, inflight, completed.
+ */
+public class ActiveInstant implements Serializable, Comparable<ActiveInstant> {

Review Comment:
   rename to "ActiveAction" since its really the instants that make up the action to completed state?



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>

Review Comment:
   can't we just read the latest manifest file? why do we want a separate version file? What does rocksdb layout look on storage? Can we learn from that.



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.

Review Comment:
   I think we can use OCC here for concurrency control between LSM merge and writer? Even just taking a lock and letting write or lsm merge fail if there was sth concurrent would be ok?



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,

Review Comment:
   might be good to think about a bound here and control how LSM merge is going to be based on that. 
   I suggest . Not having more than 1GB files, to ensure merge process can run on lower end VMs/machines. 
   



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>

Review Comment:
   what does this contain? Pointer to the latest manifest file?



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>

Review Comment:
   i.e list of all files in the entire LSM. correct?



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>

Review Comment:
   Not following. are you talking about cleaning of the Hudi data table?



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>
+ * Keeps only 3 valid snapshot versions for the reader, that means, a file is kept for at lest 3 archival trigger interval, for default configuration, it is 30 instants time span,
+ * which is far longer that the archived timeline loading time.
+ *
+ * <p><h3>Instants TTL</h3></p>
+ * The timeline reader only reads instants of last 7 days. We will skip the instants from archived timeline that are generated 1 week ago.
+ *
+ * <p><h2>Timeline Refresh</h2>

Review Comment:
   fair.



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>
+ * Keeps only 3 valid snapshot versions for the reader, that means, a file is kept for at lest 3 archival trigger interval, for default configuration, it is 30 instants time span,
+ * which is far longer that the archived timeline loading time.
+ *
+ * <p><h3>Instants TTL</h3></p>

Review Comment:
   I'd prefer lazily loading it vs ignoring it completely. i.e the LSM performs well if reading is within 1 week in the past, but should be correct always. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280086564


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,

Review Comment:
   Should we do some schemaful file to store manifests? json/parquet/avro? Can we get all information that we need to plan LSM compaction/cleaning into the manifest, so that we are not listing anything.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6539] New LSM tree style archived timeline [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1393515340


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -164,230 +101,32 @@ public boolean archiveIfRequired(HoodieEngineContext context, boolean acquireLoc
         // there is no owner or instant time per se for archival.
         txnManager.beginTransaction(Option.empty(), Option.empty());
       }
-      List<HoodieInstant> instantsToArchive = getInstantsToArchive().collect(Collectors.toList());
-      verifyLastMergeArchiveFilesIfNecessary(context);
+      // Sort again because the cleaning and rollback instants could break the sequence.
+      List<ActiveAction> instantsToArchive = getInstantsToArchive().sorted().collect(Collectors.toList());
       boolean success = true;
       if (!instantsToArchive.isEmpty()) {
-        this.writer = openWriter();
         LOG.info("Archiving instants " + instantsToArchive);
-        archive(context, instantsToArchive);
+        Consumer<Exception> exceptionHandler = e -> {
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw new HoodieException(e);
+          }
+        };
+        this.timelineWriter.write(instantsToArchive, Option.of(action -> deleteAnyLeftOverMarkers(context, action)), Option.of(exceptionHandler));
         LOG.info("Deleting archived instants " + instantsToArchive);
         success = deleteArchivedInstants(instantsToArchive, context);

Review Comment:
   we should, but currently the redundants left by failed deletion do not affect the correctness, the completion time would be still loaded correctly if an instant locates at both avtive and archived timelines.
   
   We need to think through the design though, it is arduous to maintain the whole multiple handling as atomic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1649253632

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9889e40cdf17f6f24ddefff010a063d4dd2c58e7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820) 
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * c281ded6d554350dfe362cce496d6d72cfe0bbbe UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1647773627

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799) 
   * 62001bbc6c5d9306f95d4bca3dd2bada3ca5c898 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280082412


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.

Review Comment:
   File a JIRA to track?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1282601272


##########
hudi-common/src/main/java/org/apache/hudi/common/util/ArchivedInstantReadSchemas.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.avro.Schema;
+
+/**
+ * Avro schema for different archived instant read cases.
+ */
+public abstract class ArchivedInstantReadSchemas {
+  public static final Schema SLIM_SCHEMA = new Schema.Parser().parse("{\n"

Review Comment:
   rename: TIMELINE_LSM_SLIM_READ_SCHEMA



##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java:
##########
@@ -846,6 +846,50 @@ public static List<FileStatus> getFileStatusAtLevel(
     return result;
   }
 
+  /**
+   * Renames the source file to destination, using the provided file system. If the rename failed,
+   * an attempt will be made to delete the source file.
+   */
+  public static void renameFile(FileSystem fs, Path src, Path dst) {
+    try {
+      if (fs.exists(dst)) {
+        throw new HoodieIOException("Target file " + dst + " already exists");
+      }
+      if (!fs.rename(src, dst)) {
+        handleException(fs, src, dst);
+      }
+    } catch (IOException e) {
+      handleException(fs, src, dst);
+    }
+  }
+
+  /**
+   * Handles the exception for renaming.
+   */
+  private static void handleException(FileSystem fs, Path src, Path dst) {
+    HoodieIOException ioe = new HoodieIOException("Failed to rename " + src + " to " + dst);
+    IOException ex = tryDelete(fs, src);
+    if (ex != null) {
+      ioe.addSuppressed(ex);
+    }
+    throw ioe;
+  }
+
+  /**
+   * Deletes the file from the file system.
+   *
+   * @param path the file to be deleted.
+   * @return IOException caught, if any. Null otherwise.
+   */
+  private static IOException tryDelete(FileSystem fs, Path path) {

Review Comment:
   please use `Option`



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java:
##########
@@ -552,30 +284,22 @@ private Stream<HoodieInstant> getInstantsToArchive() throws IOException {
       }
     }
 
-    return instants.flatMap(hoodieInstant -> {
+    return instants.map(hoodieInstant -> {
       List<HoodieInstant> instantsToStream = groupByTsAction.get(Pair.of(hoodieInstant.getTimestamp(),
           HoodieInstant.getComparableAction(hoodieInstant.getAction())));
-      if (instantsToStream != null) {
-        return instantsToStream.stream();
-      } else {
-        // if a concurrent writer archived the instant
-        return Stream.empty();
-      }
+      return ActiveInstant.fromInstants(instantsToStream);
     });
   }
 
-  private boolean deleteArchivedInstants(List<HoodieInstant> archivedInstants, HoodieEngineContext context) throws IOException {
+  private boolean deleteArchivedInstants(List<ActiveInstant> archivedInstants, HoodieEngineContext context) {
     LOG.info("Deleting instants " + archivedInstants);
 
     List<HoodieInstant> pendingInstants = new ArrayList<>();
     List<HoodieInstant> completedInstants = new ArrayList<>();
 
-    for (HoodieInstant instant : archivedInstants) {
-      if (instant.isCompleted()) {
-        completedInstants.add(instant);
-      } else {
-        pendingInstants.add(instant);
-      }
+    for (ActiveInstant triple : archivedInstants) {

Review Comment:
   I understand why you call this a triple. But rename to be closer to class name? '



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/ArchivedTimelineReadBenchmark.scala:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.DummyActiveInstant
+import org.apache.hudi.client.common.HoodieJavaEngineContext
+import org.apache.hudi.client.utils.{ActiveInstant, ArchivedTimelineWriter}
+import org.apache.hudi.common.model.{HoodieAvroPayload, HoodieTableType, WriteOperationType}
+import org.apache.hudi.common.table.timeline.{HoodieArchivedTimeline, HoodieInstant}
+import org.apache.hudi.common.testutils.{HoodieTestTable, HoodieTestUtils}
+import org.apache.hudi.config.{HoodieIndexConfig, HoodieWriteConfig}
+import org.apache.hudi.index.HoodieIndex.IndexType
+import org.apache.hudi.table.HoodieJavaTable
+import org.apache.spark.hudi.benchmark.{HoodieBenchmark, HoodieBenchmarkBase}
+
+import java.util
+import scala.collection.JavaConverters._
+
+object ArchivedTimelineReadBenchmark extends HoodieBenchmarkBase {

Review Comment:
   is this a jmh benchmark?



##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieArchivedManifest.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.JsonUtils;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Manifest entry for a version snapshot of the archived timeline.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class HoodieArchivedManifest implements Serializable {

Review Comment:
   lets add UTs for this class?



##########
hudi-common/src/main/java/org/apache/hudi/common/util/ArchivedInstantReadSchemas.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.avro.Schema;
+
+/**
+ * Avro schema for different archived instant read cases.
+ */
+public abstract class ArchivedInstantReadSchemas {
+  public static final Schema SLIM_SCHEMA = new Schema.Parser().parse("{\n"
+      + "   \"type\":\"record\",\n"
+      + "   \"name\":\"HoodieArchivedMetaEntryV2\",\n"
+      + "   \"namespace\":\"org.apache.hudi.avro.model\",\n"
+      + "   \"fields\":[\n"
+      + "      {\n"
+      + "         \"name\":\"instantTime\",\n"
+      + "         \"type\":[\"null\",\"string\"],\n"
+      + "         \"default\": null\n"
+      + "      },\n"
+      + "      {\n"
+      + "         \"name\":\"completionTime\",\n"
+      + "         \"type\":[\"null\",\"string\"],\n"
+      + "         \"default\": null\n"
+      + "      },\n"
+      + "      {\n"
+      + "         \"name\":\"action\",\n"
+      + "         \"type\":[\"null\",\"string\"],\n"
+      + "         \"default\": null\n"
+      + "      }\n"
+      + "   ]\n"
+      + "}");
+
+  public static final Schema SCHEMA_WITH_COMMIT_META = new Schema.Parser().parse("{\n"
+      + "   \"type\":\"record\",\n"
+      + "   \"name\":\"HoodieArchivedMetaEntryV2\",\n"
+      + "   \"namespace\":\"org.apache.hudi.avro.model\",\n"
+      + "   \"fields\":[\n"
+      + "      {\n"
+      + "         \"name\":\"instantTime\",\n"
+      + "         \"type\":[\"null\",\"string\"],\n"
+      + "         \"default\": null\n"
+      + "      },\n"
+      + "      {\n"
+      + "         \"name\":\"completionTime\",\n"
+      + "         \"type\":[\"null\",\"string\"],\n"
+      + "         \"default\": null\n"
+      + "      },\n"
+      + "      {\n"
+      + "         \"name\":\"action\",\n"
+      + "         \"type\":[\"null\",\"string\"],\n"
+      + "         \"default\": null\n"
+      + "      },\n"
+      + "      {\n"
+      + "         \"name\":\"metadata\",\n"
+      + "         \"type\":[\"null\", \"bytes\"],\n"
+      + "         \"default\": null\n"
+      + "      }\n"
+      + "   ]\n"
+      + "}");
+
+  public static final Schema SCHEMA_WITH_PLAN = new Schema.Parser().parse("{\n"

Review Comment:
   rename: TIMELINE_LSM_READ_SCHEMA_WITH_PLAN 



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  private final Map<String, byte[]> readCommits;
+
+  public LegacyArchivedMetaEntryReader(HoodieTableMetaClient metaClient) {
+    this.metaClient = metaClient;
+    this.readCommits = new HashMap<>();
+  }
+
+  /**
+   * @deprecated Use {@code HoodieArchivedTimeline#readCommit} instead.
+   */
+  private HoodieInstant readCommit(GenericRecord record, boolean loadDetails) {
+    final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
+    final String action = record.get(ACTION_TYPE_KEY).toString();
+    final String stateTransitionTime = (String) record.get(STATE_TRANSITION_TIME);
+    if (loadDetails) {
+      getMetadataKey(action).map(key -> {
+        Object actionData = record.get(key);
+        if (actionData != null) {
+          if (action.equals(HoodieTimeline.COMPACTION_ACTION)) {
+            this.readCommits.put(instantTime, HoodieAvroUtils.indexedRecordToBytes((IndexedRecord) actionData));
+          } else {
+            this.readCommits.put(instantTime, actionData.toString().getBytes(StandardCharsets.UTF_8));
+          }
+        }
+        return null;
+      });
+    }
+    return new HoodieInstant(HoodieInstant.State.valueOf(record.get(ACTION_STATE).toString()), action,
+        instantTime, stateTransitionTime);
+  }
+
+  @Nonnull
+  private Option<String> getMetadataKey(String action) {
+    switch (action) {
+      case HoodieTimeline.CLEAN_ACTION:
+        return Option.of("hoodieCleanMetadata");
+      case HoodieTimeline.COMMIT_ACTION:
+      case HoodieTimeline.DELTA_COMMIT_ACTION:
+        return Option.of("hoodieCommitMetadata");
+      case HoodieTimeline.ROLLBACK_ACTION:
+        return Option.of("hoodieRollbackMetadata");
+      case HoodieTimeline.SAVEPOINT_ACTION:
+        return Option.of("hoodieSavePointMetadata");
+      case HoodieTimeline.COMPACTION_ACTION:
+      case HoodieTimeline.LOG_COMPACTION_ACTION:
+        return Option.of("hoodieCompactionPlan");
+      case HoodieTimeline.REPLACE_COMMIT_ACTION:
+        return Option.of("hoodieReplaceCommitMetadata");
+      case HoodieTimeline.INDEXING_ACTION:
+        return Option.of("hoodieIndexCommitMetadata");
+      default:
+        LOG.error(String.format("Unknown action in metadata (%s)", action));
+        return Option.empty();
+    }
+  }
+
+  /**
+   * This is method to read selected instants. Do NOT use this directly use one of the helper methods above
+   * If loadInstantDetails is set to true, this would also update 'readCommits' map with commit details
+   * If filter is specified, only the filtered instants are loaded
+   * If commitsFilter is specified, only the filtered records are loaded
+   *
+   * @deprecated Use {@code HoodieArchivedTimeline.loadInstantsV2} instead.

Review Comment:
   I am a little lost. why do we need v1 v2 etc for new functionality? whats loadInstantsV2?



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -184,203 +222,327 @@ public HoodieArchivedTimeline reload() {
     return new HoodieArchivedTimeline(metaClient);
   }
 
-  private HoodieInstant readCommit(GenericRecord record, boolean loadDetails) {
-    final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
-    final String action = record.get(ACTION_TYPE_KEY).toString();
-    final String stateTransitionTime = (String) record.get(STATE_TRANSITION_TIME);
-    if (loadDetails) {
-      getMetadataKey(action).map(key -> {
-        Object actionData = record.get(key);
-        if (actionData != null) {
-          if (action.equals(HoodieTimeline.COMPACTION_ACTION)) {
-            this.readCommits.put(instantTime, HoodieAvroUtils.indexedRecordToBytes((IndexedRecord) actionData));
-          } else {
-            this.readCommits.put(instantTime, actionData.toString().getBytes(StandardCharsets.UTF_8));
-          }
+  private HoodieInstant readCommit(GenericRecord record, LoadMode loadMode) {
+    final String instantTime = record.get(INSTANT_TIME_ARCHIVED_META_FIELD).toString();
+    final String action = record.get(ACTION_ARCHIVED_META_FIELD).toString();
+    final String completionTime = record.get(COMPLETION_TIME_ARCHIVED_META_FIELD).toString();
+    loadInstantDetails(record, instantTime, loadMode);
+    return new HoodieInstant(HoodieInstant.State.COMPLETED, action, instantTime, completionTime);
+  }
+
+  private void loadInstantDetails(GenericRecord record, String instantTime, LoadMode loadMode) {
+    switch (loadMode) {
+      case COMMIT_META:
+        ByteBuffer commitMeta = (ByteBuffer) record.get(METADATA_ARCHIVED_META_FIELD);
+        if (commitMeta != null) {
+          // in case the entry comes from an empty completed meta file
+          this.readCommits.put(instantTime, commitMeta.array());
         }
-        return null;
-      });
-    }
-    return new HoodieInstant(HoodieInstant.State.valueOf(record.get(ACTION_STATE).toString()), action,
-        instantTime, stateTransitionTime);
-  }
-
-  @Nonnull
-  private Option<String> getMetadataKey(String action) {
-    switch (action) {
-      case HoodieTimeline.CLEAN_ACTION:
-        return Option.of("hoodieCleanMetadata");
-      case HoodieTimeline.COMMIT_ACTION:
-      case HoodieTimeline.DELTA_COMMIT_ACTION:
-        return Option.of("hoodieCommitMetadata");
-      case HoodieTimeline.ROLLBACK_ACTION:
-        return Option.of("hoodieRollbackMetadata");
-      case HoodieTimeline.SAVEPOINT_ACTION:
-        return Option.of("hoodieSavePointMetadata");
-      case HoodieTimeline.COMPACTION_ACTION:
-      case HoodieTimeline.LOG_COMPACTION_ACTION:
-        return Option.of("hoodieCompactionPlan");
-      case HoodieTimeline.REPLACE_COMMIT_ACTION:
-        return Option.of("hoodieReplaceCommitMetadata");
-      case HoodieTimeline.INDEXING_ACTION:
-        return Option.of("hoodieIndexCommitMetadata");
+        break;
+      case PLAN:
+        ByteBuffer plan = (ByteBuffer) record.get(PLAN_ARCHIVED_META_FIELD);
+        if (plan != null) {

Review Comment:
   so we may not have a plan in some cases right.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {

Review Comment:
   rename: LSMTimelineWriter



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java:
##########
@@ -94,63 +67,26 @@ public class HoodieTimelineArchiver<T extends HoodieAvroPayload, I, K, O> {
 
   private static final Logger LOG = LoggerFactory.getLogger(HoodieTimelineArchiver.class);
 
-  private final Path archiveFilePath;
   private final HoodieWriteConfig config;
-  private Writer writer;
   private final int maxInstantsToKeep;
   private final int minInstantsToKeep;
   private final HoodieTable<T, I, K, O> table;
   private final HoodieTableMetaClient metaClient;
   private final TransactionManager txnManager;
 
+  private final ArchivedTimelineWriter timelineWriter;

Review Comment:
   archivedTimelineWriter?
   Rename?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),

Review Comment:
   asserts on `instants.size()`?



##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieArchivedManifest.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.JsonUtils;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Manifest entry for a version snapshot of the archived timeline.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class HoodieArchivedManifest implements Serializable {
+  private static final Logger LOG = LoggerFactory.getLogger(HoodieArchivedManifest.class);
+
+  public static final HoodieArchivedManifest EMPTY = new HoodieArchivedManifest();
+
+  private final List<FileEntry> files;
+
+  // for ser/deser
+  public HoodieArchivedManifest() {
+    this.files = new ArrayList<>();
+  }
+
+  public HoodieArchivedManifest(List<FileEntry> files) {
+    this.files = files;
+  }
+
+  public void addFile(String fileName, long fileLen) {
+    this.files.add(FileEntry.getInstance(fileName, fileLen));
+  }
+
+  public void addFile(FileEntry fileEntry) {
+    this.files.add(fileEntry);
+  }
+
+  public List<FileEntry> getFiles() {
+    return files;
+  }
+
+  public List<String> getFileNames() {
+    return files.stream().map(FileEntry::getFileName).collect(Collectors.toList());
+  }
+
+  // -------------------------------------------------------------------------
+  //  Utilities
+  // -------------------------------------------------------------------------
+
+  public String toJsonString() throws IOException {
+    return JsonUtils.getObjectMapper().writerWithDefaultPrettyPrinter().writeValueAsString(this);
+  }
+
+  public static <T> T fromJsonString(String jsonStr, Class<T> clazz) throws Exception {
+    if (jsonStr == null || jsonStr.isEmpty()) {
+      // For empty commit file (no data or something bad happen).
+      return clazz.newInstance();
+    }
+    return JsonUtils.getObjectMapper().readValue(jsonStr, clazz);
+  }
+
+  public HoodieArchivedManifest copy(List<String> filesToRemove) {
+    List<FileEntry> newFiles = this.files.stream().filter(fileEntry -> !filesToRemove.contains(fileEntry.getFileName())).collect(Collectors.toList());
+    return new HoodieArchivedManifest(newFiles);
+  }
+
+  public HoodieArchivedManifest copy() {
+    return new HoodieArchivedManifest(new ArrayList<>(this.files));
+  }
+
+  // -------------------------------------------------------------------------
+  //  Inner Class
+  // -------------------------------------------------------------------------
+
+  /**
+   * A file entry.
+   */
+  public static class FileEntry implements Serializable, Comparable<FileEntry> {

Review Comment:
   rename: LSMFileEntry



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java:
##########
@@ -164,230 +100,27 @@ public boolean archiveIfRequired(HoodieEngineContext context, boolean acquireLoc
         // there is no owner or instant time per se for archival.
         txnManager.beginTransaction(Option.empty(), Option.empty());
       }
-      List<HoodieInstant> instantsToArchive = getInstantsToArchive().collect(Collectors.toList());
-      verifyLastMergeArchiveFilesIfNecessary(context);
+      // Sort again because the cleaning and rollback instants could break the sequence.
+      List<ActiveInstant> instantsToArchive = getInstantsToArchive().sorted().collect(Collectors.toList());
       boolean success = true;
       if (!instantsToArchive.isEmpty()) {
-        this.writer = openWriter();
         LOG.info("Archiving instants " + instantsToArchive);
-        archive(context, instantsToArchive);
+        this.timelineWriter.write(context, instantsToArchive);
         LOG.info("Deleting archived instants " + instantsToArchive);
         success = deleteArchivedInstants(instantsToArchive, context);
+        // triggers compaction and cleaning only after archiving action
+        this.timelineWriter.compactAndClean(context);
       } else {
         LOG.info("No Instants to archive");
       }
-
-      if (shouldMergeSmallArchiveFiles()) {

Review Comment:
   Can you confirm we are removing this functionality fully within this pull request?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {

Review Comment:
   want just to make both arguments as lists? I can imagine compaction adding multiple files. we can reuse the same method there too.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java:
##########
@@ -603,53 +327,4 @@ private boolean deleteArchivedInstants(List<HoodieInstant> archivedInstants, Hoo
 
     return true;
   }
-
-  public void archive(HoodieEngineContext context, List<HoodieInstant> instants) throws HoodieCommitException {
-    try {
-      Schema wrapperSchema = HoodieArchivedMetaEntry.getClassSchema();
-      LOG.info("Wrapper schema " + wrapperSchema.toString());
-      List<IndexedRecord> records = new ArrayList<>();
-      for (HoodieInstant hoodieInstant : instants) {
-        try {
-          deleteAnyLeftOverMarkers(context, hoodieInstant);
-          records.add(convertToAvroRecord(hoodieInstant));
-          if (records.size() >= this.config.getCommitArchivalBatchSize()) {
-            writeToFile(wrapperSchema, records);
-          }
-        } catch (Exception e) {
-          LOG.error("Failed to archive commits, .commit file: " + hoodieInstant.getFileName(), e);
-          if (this.config.isFailOnTimelineArchivingEnabled()) {
-            throw e;
-          }
-        }
-      }
-      writeToFile(wrapperSchema, records);
-    } catch (Exception e) {
-      throw new HoodieCommitException("Failed to archive commits", e);
-    }
-  }
-
-  private void deleteAnyLeftOverMarkers(HoodieEngineContext context, HoodieInstant instant) {

Review Comment:
   Are these moved over?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java:
##########
@@ -493,17 +224,18 @@ private Stream<HoodieInstant> getCommitInstantsToArchive() throws IOException {
     }
   }
 
-  private Stream<HoodieInstant> getInstantsToArchive() throws IOException {
-    Stream<HoodieInstant> instants = Stream.concat(getCleanInstantsToArchive(), getCommitInstantsToArchive());
+  private Stream<ActiveInstant> getInstantsToArchive() throws IOException {

Review Comment:
   Can you confirm no logic is changing around choosing the instants we archive?
   



##########
hudi-common/src/main/java/org/apache/hudi/common/util/ArchivedInstantReadSchemas.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.avro.Schema;
+
+/**
+ * Avro schema for different archived instant read cases.
+ */
+public abstract class ArchivedInstantReadSchemas {

Review Comment:
   move these schemas to .avsc files? or build programmatically?



##########
hudi-common/src/main/java/org/apache/hudi/common/util/ArchivedInstantReadSchemas.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.avro.Schema;
+
+/**
+ * Avro schema for different archived instant read cases.
+ */
+public abstract class ArchivedInstantReadSchemas {
+  public static final Schema SLIM_SCHEMA = new Schema.Parser().parse("{\n"
+      + "   \"type\":\"record\",\n"
+      + "   \"name\":\"HoodieArchivedMetaEntryV2\",\n"
+      + "   \"namespace\":\"org.apache.hudi.avro.model\",\n"
+      + "   \"fields\":[\n"
+      + "      {\n"
+      + "         \"name\":\"instantTime\",\n"
+      + "         \"type\":[\"null\",\"string\"],\n"
+      + "         \"default\": null\n"
+      + "      },\n"
+      + "      {\n"
+      + "         \"name\":\"completionTime\",\n"
+      + "         \"type\":[\"null\",\"string\"],\n"
+      + "         \"default\": null\n"
+      + "      },\n"
+      + "      {\n"
+      + "         \"name\":\"action\",\n"
+      + "         \"type\":[\"null\",\"string\"],\n"
+      + "         \"default\": null\n"
+      + "      }\n"
+      + "   ]\n"
+      + "}");
+
+  public static final Schema SCHEMA_WITH_COMMIT_META = new Schema.Parser().parse("{\n"

Review Comment:
   rename: TIMELINE_LSM_READ_SCHEMA_WITH_METADATA (not every instant is a commit, so please avoid generalizing that way)



##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieArchivedManifest.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.JsonUtils;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Manifest entry for a version snapshot of the archived timeline.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class HoodieArchivedManifest implements Serializable {

Review Comment:
   rename: HoodieLSMTimelineManifest 
   
   Try renaming everything more generally? so when we try to break active vs archived differences, we don't have to rename everything again. 



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {
+      // 2. sort files by min instant time (implies ascending chronological order)
+      files.sort(HoodieArchivedManifest.FileEntry::compareTo);
+      List<String> candidateFiles = getCandidateFiles(files, compactionBatchSize);
+      if (candidateFiles.size() < 2) {
+        // the file is too large to compact, returns early.
+        return Option.empty();
+      }
+      String compactedFileName = compactedFileName(candidateFiles);
+
+      // 3. compaction
+      compactArchiveFiles(candidateFiles, compactedFileName);
+      // 4. update the manifest file
+      updateManifest(candidateFiles, compactedFileName);
+      LOG.info("Finishes compaction of archive files: " + candidateFiles);
+      return Option.of(compactedFileName);
+    }
+    return Option.empty();
+  }
+
+  public void compactArchiveFiles(List<String> candidateFiles, String compactedFileName) {

Review Comment:
   rename.w/o archive in the name



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {
+      // 2. sort files by min instant time (implies ascending chronological order)
+      files.sort(HoodieArchivedManifest.FileEntry::compareTo);
+      List<String> candidateFiles = getCandidateFiles(files, compactionBatchSize);
+      if (candidateFiles.size() < 2) {
+        // the file is too large to compact, returns early.
+        return Option.empty();
+      }
+      String compactedFileName = compactedFileName(candidateFiles);
+
+      // 3. compaction
+      compactArchiveFiles(candidateFiles, compactedFileName);
+      // 4. update the manifest file
+      updateManifest(candidateFiles, compactedFileName);
+      LOG.info("Finishes compaction of archive files: " + candidateFiles);
+      return Option.of(compactedFileName);
+    }
+    return Option.empty();
+  }
+
+  public void compactArchiveFiles(List<String> candidateFiles, String compactedFileName) {
+    LOG.info("Starting to merge small archive files.");
+    try (HoodieFileWriter writer = openWriter(new Path(metaClient.getArchivePath(), compactedFileName))) {

Review Comment:
   Does this code implement a N-way sort-merge from candidateFiles to target? is n't writing one-by-one, so the entries inside are not sorted? I wrote some code many years ago for sth like that. https://github.com/vinothchandar/hudi/commit/9050dd1317851bc395aef51ab60d761f74c0529d



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {

Review Comment:
   this needs to be total bytes or entries based?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {
+      // 2. sort files by min instant time (implies ascending chronological order)
+      files.sort(HoodieArchivedManifest.FileEntry::compareTo);
+      List<String> candidateFiles = getCandidateFiles(files, compactionBatchSize);
+      if (candidateFiles.size() < 2) {
+        // the file is too large to compact, returns early.
+        return Option.empty();
+      }
+      String compactedFileName = compactedFileName(candidateFiles);
+
+      // 3. compaction
+      compactArchiveFiles(candidateFiles, compactedFileName);
+      // 4. update the manifest file
+      updateManifest(candidateFiles, compactedFileName);
+      LOG.info("Finishes compaction of archive files: " + candidateFiles);
+      return Option.of(compactedFileName);
+    }
+    return Option.empty();
+  }
+
+  public void compactArchiveFiles(List<String> candidateFiles, String compactedFileName) {
+    LOG.info("Starting to merge small archive files.");
+    try (HoodieFileWriter writer = openWriter(new Path(metaClient.getArchivePath(), compactedFileName))) {
+      for (String fileName : candidateFiles) {
+        // Read the archived file
+        try (HoodieAvroParquetReader reader = (HoodieAvroParquetReader) HoodieFileReaderFactory.getReaderFactory(HoodieRecord.HoodieRecordType.AVRO)
+            .getFileReader(metaClient.getHadoopConf(), new Path(metaClient.getArchivePath(), fileName))) {
+          // Read the meta entry
+          try (ClosableIterator<IndexedRecord> iterator = reader.getIndexedRecordIterator(HoodieArchivedInstant.getClassSchema(), HoodieArchivedInstant.getClassSchema())) {
+            while (iterator.hasNext()) {
+              IndexedRecord record = iterator.next();
+              writer.write(record.get(0).toString(), new HoodieAvroIndexedRecord(record), HoodieArchivedInstant.getClassSchema());
+            }
+          }
+        }
+      }
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to merge small archive files", e);
+    }
+    LOG.info("Success to merge small archive files.");
+  }
+
+  /**
+   * Checks whether there is any unfinished compaction operation.
+   *
+   * @param context HoodieEngineContext used for parallelize to delete small archive files if necessary.
+   */
+  public void clean(HoodieEngineContext context, int compactedVersions) throws IOException {
+    // if there are more than 3 version of snapshots, clean the oldest files.
+    List<Integer> allSnapshotVersions = HoodieArchivedTimeline.allSnapshotVersions(metaClient);
+    int numVersionsToKeep = 3 + compactedVersions; // should make the threshold configurable.
+    if (allSnapshotVersions.size() > numVersionsToKeep) {
+      allSnapshotVersions.sort((v1, v2) -> v2 - v1);
+      List<Integer> versionsToKeep = allSnapshotVersions.subList(0, numVersionsToKeep);
+      Set<String> filesToKeep = versionsToKeep.stream()
+          .flatMap(version -> HoodieArchivedTimeline.latestSnapshotManifest(metaClient, version).getFileNames().stream())
+          .collect(Collectors.toSet());
+      // delete the manifest file first
+      List<String> manifestFilesToClean = new ArrayList<>();
+      Arrays.stream(HoodieArchivedTimeline.listAllManifestFiles(metaClient)).forEach(fileStatus -> {
+        if (!versionsToKeep.contains(HoodieArchivedTimeline.getManifestVersion(fileStatus.getPath().getName()))) {
+          manifestFilesToClean.add(fileStatus.getPath().toString());
+        }
+      });
+      deleteFilesParallelize(metaClient, manifestFilesToClean, context, false);

Review Comment:
   what if we crash after this. 



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  private final Map<String, byte[]> readCommits;
+
+  public LegacyArchivedMetaEntryReader(HoodieTableMetaClient metaClient) {
+    this.metaClient = metaClient;
+    this.readCommits = new HashMap<>();
+  }
+
+  /**
+   * @deprecated Use {@code HoodieArchivedTimeline#readCommit} instead.

Review Comment:
   Is the comment ok? Or needs to be changed?



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/FileCreateUtils.java:
##########
@@ -278,29 +278,16 @@ public static void createRestoreFile(String basePath, String instantTime, Hoodie
     createMetaFile(basePath, instantTime, HoodieTimeline.RESTORE_ACTION, serializeRestoreMetadata(hoodieRestoreMetadata).get());
   }
 
-  private static void createAuxiliaryMetaFile(String basePath, String instantTime, String suffix) throws IOException {

Review Comment:
   note to self: we don't need the aux meta files anymore?



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,127 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
+import java.nio.ByteBuffer;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
-import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *     \              /
+ *        \        /
+ *            |
+ *            V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *                                      |
+ *                                      V
+ *                                  _version_
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a compacted file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.

Review Comment:
   move all this out to LSMTimeline class.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  private final Map<String, byte[]> readCommits;
+
+  public LegacyArchivedMetaEntryReader(HoodieTableMetaClient metaClient) {
+    this.metaClient = metaClient;
+    this.readCommits = new HashMap<>();
+  }
+
+  /**
+   * @deprecated Use {@code HoodieArchivedTimeline#readCommit} instead.
+   */
+  private HoodieInstant readCommit(GenericRecord record, boolean loadDetails) {

Review Comment:
   rename: readInstant



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,127 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
+import java.nio.ByteBuffer;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
-import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.

Review Comment:
   please this class free from LSM design. Single Responsibility Principle



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {

Review Comment:
   lets write a test for this class?



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/io/TestHoodieTimelineArchiver.java:
##########
@@ -502,38 +493,26 @@ public void testSavepointWithArchival(boolean archiveBeyondSavepoint) throws Exc
         getActiveCommitInstants(Arrays.asList("00000008", "00000009")), commitsAfterArchival);
   }
 
-  @ParameterizedTest
-  @ValueSource(booleans = {true, false})
-  public void testMergeSmallArchiveFilesRecoverFromBuildPlanFailed(boolean enableArchiveMerge) throws Exception {
-    HoodieWriteConfig writeConfig = initTestTableAndGetWriteConfig(true, 4, 5, 2, enableArchiveMerge, 3, 209715200);
+  @Test
+  public void testCompactionWithCorruptVersionFile() throws Exception {
+    HoodieWriteConfig writeConfig = initTestTableAndGetWriteConfig(true, 4, 5, 2, 3);
 
     // do ingestion and trigger archive actions here.
     for (int i = 1; i < 10; i++) {
       testTable.doWriteOperation("0000000" + i, WriteOperationType.UPSERT, i == 1 ? Arrays.asList("p1", "p2") : Collections.emptyList(), Arrays.asList("p1", "p2"), 2);
       archiveAndGetCommitsList(writeConfig);
     }
 
-    // build a merge small archive plan with dummy content

Review Comment:
   I see. you are replacing the existing small archived file merging impl with the new LSM based one



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  private final Map<String, byte[]> readCommits;

Review Comment:
   rename: readInstants



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {
+      // 2. sort files by min instant time (implies ascending chronological order)
+      files.sort(HoodieArchivedManifest.FileEntry::compareTo);
+      List<String> candidateFiles = getCandidateFiles(files, compactionBatchSize);
+      if (candidateFiles.size() < 2) {
+        // the file is too large to compact, returns early.
+        return Option.empty();
+      }
+      String compactedFileName = compactedFileName(candidateFiles);
+
+      // 3. compaction
+      compactArchiveFiles(candidateFiles, compactedFileName);
+      // 4. update the manifest file
+      updateManifest(candidateFiles, compactedFileName);
+      LOG.info("Finishes compaction of archive files: " + candidateFiles);
+      return Option.of(compactedFileName);
+    }
+    return Option.empty();
+  }
+
+  public void compactArchiveFiles(List<String> candidateFiles, String compactedFileName) {
+    LOG.info("Starting to merge small archive files.");
+    try (HoodieFileWriter writer = openWriter(new Path(metaClient.getArchivePath(), compactedFileName))) {
+      for (String fileName : candidateFiles) {
+        // Read the archived file
+        try (HoodieAvroParquetReader reader = (HoodieAvroParquetReader) HoodieFileReaderFactory.getReaderFactory(HoodieRecord.HoodieRecordType.AVRO)
+            .getFileReader(metaClient.getHadoopConf(), new Path(metaClient.getArchivePath(), fileName))) {
+          // Read the meta entry
+          try (ClosableIterator<IndexedRecord> iterator = reader.getIndexedRecordIterator(HoodieArchivedInstant.getClassSchema(), HoodieArchivedInstant.getClassSchema())) {
+            while (iterator.hasNext()) {
+              IndexedRecord record = iterator.next();
+              writer.write(record.get(0).toString(), new HoodieAvroIndexedRecord(record), HoodieArchivedInstant.getClassSchema());
+            }
+          }
+        }
+      }
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to merge small archive files", e);
+    }
+    LOG.info("Success to merge small archive files.");
+  }
+
+  /**
+   * Checks whether there is any unfinished compaction operation.
+   *
+   * @param context HoodieEngineContext used for parallelize to delete small archive files if necessary.
+   */
+  public void clean(HoodieEngineContext context, int compactedVersions) throws IOException {
+    // if there are more than 3 version of snapshots, clean the oldest files.
+    List<Integer> allSnapshotVersions = HoodieArchivedTimeline.allSnapshotVersions(metaClient);
+    int numVersionsToKeep = 3 + compactedVersions; // should make the threshold configurable.
+    if (allSnapshotVersions.size() > numVersionsToKeep) {
+      allSnapshotVersions.sort((v1, v2) -> v2 - v1);
+      List<Integer> versionsToKeep = allSnapshotVersions.subList(0, numVersionsToKeep);
+      Set<String> filesToKeep = versionsToKeep.stream()
+          .flatMap(version -> HoodieArchivedTimeline.latestSnapshotManifest(metaClient, version).getFileNames().stream())
+          .collect(Collectors.toSet());
+      // delete the manifest file first
+      List<String> manifestFilesToClean = new ArrayList<>();
+      Arrays.stream(HoodieArchivedTimeline.listAllManifestFiles(metaClient)).forEach(fileStatus -> {
+        if (!versionsToKeep.contains(HoodieArchivedTimeline.getManifestVersion(fileStatus.getPath().getName()))) {
+          manifestFilesToClean.add(fileStatus.getPath().toString());
+        }
+      });
+      deleteFilesParallelize(metaClient, manifestFilesToClean, context, false);
+      // delete the archive data files
+      List<String> dataFilesToClean = Arrays.stream(HoodieArchivedTimeline.listAllMetaFiles(metaClient))
+          .filter(fileStatus -> !filesToKeep.contains(fileStatus.getPath().getName()))
+          .map(fileStatus -> fileStatus.getPath().toString())
+          .collect(Collectors.toList());
+      deleteFilesParallelize(metaClient, dataFilesToClean, context, false);
+    }
+  }
+
+  private HoodieArchivedManifest.FileEntry getFileEntry(String fileName) throws IOException {
+    long fileLen = metaClient.getFs().getFileStatus(new Path(metaClient.getArchivePath(), fileName)).getLen();
+    return HoodieArchivedManifest.FileEntry.getInstance(fileName, fileLen);
+  }
+
+  /**
+   * Returns at most {@code filesBatch} number of source files
+   * restricted by the gross file size by 1GB.
+   */
+  private List<String> getCandidateFiles(List<HoodieArchivedManifest.FileEntry> files, int filesBatch) throws IOException {
+    List<String> candidates = new ArrayList<>();
+    long totalFileLen = 0L;
+    long maxFileSizeInBytes = 1024 * 1024 * 1000;
+    for (int i = 0; i < filesBatch; i++) {
+      HoodieArchivedManifest.FileEntry fileEntry = files.get(i);
+      if (totalFileLen > maxFileSizeInBytes) {
+        return candidates;
+      }
+      // we may also need to consider a single file that is very close to the threshold in size,
+      // to avoid the write amplification,
+      // for e.g, two 800MB files compact into a 1.6GB file.
+      totalFileLen += fileEntry.getFileLen();
+      candidates.add(fileEntry.getFileName());
+    }
+    return candidates;
+  }
+
+  /**
+   * Returns a new file name.
+   */
+  private static String newFileName(String minInstant, String maxInstant, int layer) {
+    return minInstant + "_" + maxInstant + "_" + layer + HoodieFileFormat.PARQUET.getFileExtension();
+  }
+
+  /**
+   * Returns a new file name.
+   */
+  @VisibleForTesting
+  public static String compactedFileName(List<String> files) {
+    String minInstant = files.stream().map(HoodieArchivedTimeline::getMinInstantTime)
+        .min(Comparator.naturalOrder()).get();
+    String maxInstant = files.stream().map(HoodieArchivedTimeline::getMaxInstantTime)
+        .max(Comparator.naturalOrder()).get();
+    int currentLayer = HoodieArchivedTimeline.getFileLayer(files.get(0));
+    return newFileName(minInstant, maxInstant, currentLayer + 1);
+  }
+
+  /**
+   * Get or create a writer config for parquet writer.
+   */
+  private HoodieWriteConfig getOrCreateWriterConfig() {
+    if (this.writeConfig == null) {
+      this.writeConfig = HoodieWriteConfig.newBuilder()
+          .withProperties(this.config.getProps())
+          .withPopulateMetaFields(false).build();
+    }
+    return this.writeConfig;
+  }
+
+  private HoodieFileWriter openWriter(Path filePath) {
+    try {
+      return HoodieFileWriterFactory.getFileWriter("", filePath, metaClient.getHadoopConf(), getOrCreateWriterConfig(),
+          HoodieArchivedInstant.getClassSchema(), table.getTaskContextSupplier(), HoodieRecord.HoodieRecordType.AVRO);
+    } catch (IOException e) {
+      throw new HoodieException("Unable to initialize archiving writer", e);
+    }
+  }
+
+  private void deleteAnyLeftOverMarkers(HoodieEngineContext context, ActiveInstant activeInstant) {
+    WriteMarkers writeMarkers = WriteMarkersFactory.get(config.getMarkersType(), table, activeInstant.getInstantTime());
+    if (writeMarkers.deleteMarkerDir(context, config.getMarkersDeleteParallelism())) {
+      LOG.info("Cleaned up left over marker directory for instant :" + activeInstant.getCompleted());
+    }
+  }
+
+  private Map<String, Boolean> deleteFilesParallelize(

Review Comment:
   move to FSUtils?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveInstant.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * A combination of instants covering action states: requested, inflight, completed.
+ */
+public class ActiveInstant implements Serializable, Comparable<ActiveInstant> {

Review Comment:
   Lets call this class. ActiveAction which is a triplet of instants . That's how we call this today 



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;

Review Comment:
   so we don't compact beyond layer 0 now?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  private final Map<String, byte[]> readCommits;

Review Comment:
   are you assuming the entire archived timeline can be read into a single jvm memory? If yes, this won't hold good for most reasonably large deployments. 
   I think we need to do a streaming read from the log files in legacy archived timeline and buffer enough to write into LSM Timeline file by file.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {
+      // 2. sort files by min instant time (implies ascending chronological order)
+      files.sort(HoodieArchivedManifest.FileEntry::compareTo);
+      List<String> candidateFiles = getCandidateFiles(files, compactionBatchSize);
+      if (candidateFiles.size() < 2) {
+        // the file is too large to compact, returns early.
+        return Option.empty();
+      }
+      String compactedFileName = compactedFileName(candidateFiles);
+
+      // 3. compaction
+      compactArchiveFiles(candidateFiles, compactedFileName);
+      // 4. update the manifest file
+      updateManifest(candidateFiles, compactedFileName);
+      LOG.info("Finishes compaction of archive files: " + candidateFiles);
+      return Option.of(compactedFileName);
+    }
+    return Option.empty();
+  }
+
+  public void compactArchiveFiles(List<String> candidateFiles, String compactedFileName) {
+    LOG.info("Starting to merge small archive files.");
+    try (HoodieFileWriter writer = openWriter(new Path(metaClient.getArchivePath(), compactedFileName))) {
+      for (String fileName : candidateFiles) {
+        // Read the archived file
+        try (HoodieAvroParquetReader reader = (HoodieAvroParquetReader) HoodieFileReaderFactory.getReaderFactory(HoodieRecord.HoodieRecordType.AVRO)
+            .getFileReader(metaClient.getHadoopConf(), new Path(metaClient.getArchivePath(), fileName))) {
+          // Read the meta entry
+          try (ClosableIterator<IndexedRecord> iterator = reader.getIndexedRecordIterator(HoodieArchivedInstant.getClassSchema(), HoodieArchivedInstant.getClassSchema())) {
+            while (iterator.hasNext()) {
+              IndexedRecord record = iterator.next();
+              writer.write(record.get(0).toString(), new HoodieAvroIndexedRecord(record), HoodieArchivedInstant.getClassSchema());
+            }
+          }
+        }
+      }
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to merge small archive files", e);
+    }
+    LOG.info("Success to merge small archive files.");
+  }
+
+  /**
+   * Checks whether there is any unfinished compaction operation.
+   *
+   * @param context HoodieEngineContext used for parallelize to delete small archive files if necessary.
+   */
+  public void clean(HoodieEngineContext context, int compactedVersions) throws IOException {
+    // if there are more than 3 version of snapshots, clean the oldest files.
+    List<Integer> allSnapshotVersions = HoodieArchivedTimeline.allSnapshotVersions(metaClient);
+    int numVersionsToKeep = 3 + compactedVersions; // should make the threshold configurable.
+    if (allSnapshotVersions.size() > numVersionsToKeep) {
+      allSnapshotVersions.sort((v1, v2) -> v2 - v1);
+      List<Integer> versionsToKeep = allSnapshotVersions.subList(0, numVersionsToKeep);
+      Set<String> filesToKeep = versionsToKeep.stream()
+          .flatMap(version -> HoodieArchivedTimeline.latestSnapshotManifest(metaClient, version).getFileNames().stream())
+          .collect(Collectors.toSet());
+      // delete the manifest file first
+      List<String> manifestFilesToClean = new ArrayList<>();
+      Arrays.stream(HoodieArchivedTimeline.listAllManifestFiles(metaClient)).forEach(fileStatus -> {
+        if (!versionsToKeep.contains(HoodieArchivedTimeline.getManifestVersion(fileStatus.getPath().getName()))) {
+          manifestFilesToClean.add(fileStatus.getPath().toString());
+        }
+      });
+      deleteFilesParallelize(metaClient, manifestFilesToClean, context, false);
+      // delete the archive data files
+      List<String> dataFilesToClean = Arrays.stream(HoodieArchivedTimeline.listAllMetaFiles(metaClient))
+          .filter(fileStatus -> !filesToKeep.contains(fileStatus.getPath().getName()))
+          .map(fileStatus -> fileStatus.getPath().toString())
+          .collect(Collectors.toList());
+      deleteFilesParallelize(metaClient, dataFilesToClean, context, false);
+    }
+  }
+
+  private HoodieArchivedManifest.FileEntry getFileEntry(String fileName) throws IOException {
+    long fileLen = metaClient.getFs().getFileStatus(new Path(metaClient.getArchivePath(), fileName)).getLen();
+    return HoodieArchivedManifest.FileEntry.getInstance(fileName, fileLen);
+  }
+
+  /**
+   * Returns at most {@code filesBatch} number of source files
+   * restricted by the gross file size by 1GB.
+   */
+  private List<String> getCandidateFiles(List<HoodieArchivedManifest.FileEntry> files, int filesBatch) throws IOException {
+    List<String> candidates = new ArrayList<>();
+    long totalFileLen = 0L;
+    long maxFileSizeInBytes = 1024 * 1024 * 1000;

Review Comment:
   pull into constant



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java:
##########
@@ -137,6 +141,41 @@ public static HoodieArchivedMetaEntry createMetaWrapper(HoodieInstant hoodieInst
     return archivedMetaWrapper;
   }
 
+  public static HoodieArchivedInstant createArchivedInstant(ActiveInstant activeInstant, HoodieTableMetaClient metaClient) {

Review Comment:
   add UT
   



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  private final Map<String, byte[]> readCommits;
+
+  public LegacyArchivedMetaEntryReader(HoodieTableMetaClient metaClient) {
+    this.metaClient = metaClient;
+    this.readCommits = new HashMap<>();
+  }
+
+  /**
+   * @deprecated Use {@code HoodieArchivedTimeline#readCommit} instead.
+   */
+  private HoodieInstant readCommit(GenericRecord record, boolean loadDetails) {
+    final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
+    final String action = record.get(ACTION_TYPE_KEY).toString();
+    final String stateTransitionTime = (String) record.get(STATE_TRANSITION_TIME);
+    if (loadDetails) {
+      getMetadataKey(action).map(key -> {
+        Object actionData = record.get(key);
+        if (actionData != null) {
+          if (action.equals(HoodieTimeline.COMPACTION_ACTION)) {
+            this.readCommits.put(instantTime, HoodieAvroUtils.indexedRecordToBytes((IndexedRecord) actionData));
+          } else {
+            this.readCommits.put(instantTime, actionData.toString().getBytes(StandardCharsets.UTF_8));
+          }
+        }
+        return null;
+      });
+    }
+    return new HoodieInstant(HoodieInstant.State.valueOf(record.get(ACTION_STATE).toString()), action,
+        instantTime, stateTransitionTime);
+  }
+
+  @Nonnull
+  private Option<String> getMetadataKey(String action) {
+    switch (action) {
+      case HoodieTimeline.CLEAN_ACTION:
+        return Option.of("hoodieCleanMetadata");
+      case HoodieTimeline.COMMIT_ACTION:
+      case HoodieTimeline.DELTA_COMMIT_ACTION:
+        return Option.of("hoodieCommitMetadata");
+      case HoodieTimeline.ROLLBACK_ACTION:
+        return Option.of("hoodieRollbackMetadata");
+      case HoodieTimeline.SAVEPOINT_ACTION:
+        return Option.of("hoodieSavePointMetadata");
+      case HoodieTimeline.COMPACTION_ACTION:
+      case HoodieTimeline.LOG_COMPACTION_ACTION:
+        return Option.of("hoodieCompactionPlan");
+      case HoodieTimeline.REPLACE_COMMIT_ACTION:
+        return Option.of("hoodieReplaceCommitMetadata");
+      case HoodieTimeline.INDEXING_ACTION:
+        return Option.of("hoodieIndexCommitMetadata");
+      default:
+        LOG.error(String.format("Unknown action in metadata (%s)", action));
+        return Option.empty();
+    }
+  }
+
+  /**
+   * This is method to read selected instants. Do NOT use this directly use one of the helper methods above
+   * If loadInstantDetails is set to true, this would also update 'readCommits' map with commit details
+   * If filter is specified, only the filtered instants are loaded
+   * If commitsFilter is specified, only the filtered records are loaded
+   *
+   * @deprecated Use {@code HoodieArchivedTimeline.loadInstantsV2} instead.
+   */
+  private List<HoodieInstant> loadInstants(
+      HoodieArchivedTimeline.TimeRangeFilter filter,
+      boolean loadInstantDetails,
+      Function<GenericRecord, Boolean> commitsFilter) {
+    try {
+      // List all files
+      FileStatus[] fsStatuses = metaClient.getFs().globStatus(
+          new Path(metaClient.getArchivePath() + "/.commits_.archive*"));
+
+      // Sort files by version suffix in reverse (implies reverse chronological order)
+      Arrays.sort(fsStatuses, new ArchiveLogVersionComparator());
+
+      Set<HoodieInstant> instantsInRange = new HashSet<>();
+      for (FileStatus fs : fsStatuses) {
+        // Read the archived file
+        try (HoodieLogFormat.Reader reader = HoodieLogFormat.newReader(metaClient.getFs(),
+            new HoodieLogFile(fs.getPath()), HoodieArchivedMetaEntry.getClassSchema())) {
+          int instantsInPreviousFile = instantsInRange.size();
+          // Read the avro blocks
+          while (reader.hasNext()) {
+            HoodieLogBlock block = reader.next();
+            if (block instanceof HoodieAvroDataBlock) {
+              HoodieAvroDataBlock avroBlock = (HoodieAvroDataBlock) block;
+              // TODO If we can store additional metadata in datablock, we can skip parsing records
+              // (such as startTime, endTime of records in the block)
+              try (ClosableIterator<HoodieRecord<IndexedRecord>> itr = avroBlock.getRecordIterator(HoodieRecord.HoodieRecordType.AVRO)) {
+                StreamSupport.stream(Spliterators.spliteratorUnknownSize(itr, Spliterator.IMMUTABLE), true)
+                    // Filter blocks in desired time window
+                    .map(r -> (GenericRecord) r.getData())
+                    .filter(commitsFilter::apply)
+                    .map(r -> readCommit(r, loadInstantDetails))
+                    .filter(c -> filter == null || filter.isInRange(c.getTimestamp()))
+                    .forEach(instantsInRange::add);
+              }
+            }
+          }
+
+          if (filter != null) {
+            int instantsInCurrentFile = instantsInRange.size() - instantsInPreviousFile;
+            if (instantsInPreviousFile > 0 && instantsInCurrentFile == 0) {
+              // Note that this is an optimization to skip reading unnecessary archived files
+              // This signals we crossed lower bound of desired time window.
+              break;
+            }
+          }
+        } catch (Exception oriException) {
+          // merge small archive files may leave uncompleted archive file which will cause exception.
+          // need to ignore this kind of exception here.
+          try {
+            if (isCorruptCompactionFile(metaClient, fs)) {
+              continue;
+            }
+            throw oriException;
+          } catch (Exception e) {
+            // If anything wrong during parsing merge archive plan, we need to throw the original exception.
+            // For example corrupted archive file and corrupted plan are both existed.
+            throw oriException;
+          }
+        }
+      }
+
+      ArrayList<HoodieInstant> result = new ArrayList<>(instantsInRange);
+      Collections.sort(result);
+      return result;
+    } catch (IOException e) {
+      throw new HoodieIOException(
+          "Could not load archived commit timeline from path " + metaClient.getArchivePath(), e);
+    }
+  }
+
+  public static boolean isCorruptCompactionFile(HoodieTableMetaClient metaClient, FileStatus fileStatus) throws IOException {
+    Path planPath = new Path(metaClient.getArchivePath(), MERGE_ARCHIVE_PLAN_NAME);
+    HoodieWrapperFileSystem fileSystem = metaClient.getFs();
+    if (fileSystem.exists(planPath)) {
+      HoodieMergeArchiveFilePlan plan = TimelineMetadataUtils.deserializeAvroMetadata(FileIOUtils.readDataFromPath(fileSystem, planPath).get(), HoodieMergeArchiveFilePlan.class);
+      String mergedArchiveFileName = plan.getMergedArchiveFileName();
+      if (!StringUtils.isNullOrEmpty(mergedArchiveFileName) && fileStatus.getPath().getName().equalsIgnoreCase(mergedArchiveFileName)) {
+        LOG.warn("Catch exception because of reading uncompleted merging archive file " + mergedArchiveFileName + ". Ignore it here.");
+        return true;
+      }
+    }
+    return false;
+  }
+
+  /**
+   * Sort files by reverse order of version suffix in file name.
+   */
+  public static class ArchiveLogVersionComparator implements Comparator<FileStatus>, Serializable {

Review Comment:
   tests



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/MetadataConversionUtils.java:
##########
@@ -137,6 +141,41 @@ public static HoodieArchivedMetaEntry createMetaWrapper(HoodieInstant hoodieInst
     return archivedMetaWrapper;
   }
 
+  public static HoodieArchivedInstant createArchivedInstant(ActiveInstant activeInstant, HoodieTableMetaClient metaClient) {
+    HoodieArchivedInstant archivedInstant = new HoodieArchivedInstant();

Review Comment:
   nts: I need to review this more closely. Seems ok at first pass



##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieArchivedManifest.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.JsonUtils;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Manifest entry for a version snapshot of the archived timeline.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class HoodieArchivedManifest implements Serializable {
+  private static final Logger LOG = LoggerFactory.getLogger(HoodieArchivedManifest.class);
+
+  public static final HoodieArchivedManifest EMPTY = new HoodieArchivedManifest();
+
+  private final List<FileEntry> files;
+
+  // for ser/deser
+  public HoodieArchivedManifest() {
+    this.files = new ArrayList<>();
+  }
+
+  public HoodieArchivedManifest(List<FileEntry> files) {
+    this.files = files;
+  }
+
+  public void addFile(String fileName, long fileLen) {
+    this.files.add(FileEntry.getInstance(fileName, fileLen));
+  }
+
+  public void addFile(FileEntry fileEntry) {
+    this.files.add(fileEntry);
+  }
+
+  public List<FileEntry> getFiles() {
+    return files;
+  }
+
+  public List<String> getFileNames() {
+    return files.stream().map(FileEntry::getFileName).collect(Collectors.toList());
+  }
+
+  // -------------------------------------------------------------------------
+  //  Utilities
+  // -------------------------------------------------------------------------
+
+  public String toJsonString() throws IOException {
+    return JsonUtils.getObjectMapper().writerWithDefaultPrettyPrinter().writeValueAsString(this);
+  }
+
+  public static <T> T fromJsonString(String jsonStr, Class<T> clazz) throws Exception {
+    if (jsonStr == null || jsonStr.isEmpty()) {
+      // For empty commit file (no data or something bad happen).
+      return clazz.newInstance();
+    }
+    return JsonUtils.getObjectMapper().readValue(jsonStr, clazz);
+  }
+
+  public HoodieArchivedManifest copy(List<String> filesToRemove) {
+    List<FileEntry> newFiles = this.files.stream().filter(fileEntry -> !filesToRemove.contains(fileEntry.getFileName())).collect(Collectors.toList());
+    return new HoodieArchivedManifest(newFiles);
+  }
+
+  public HoodieArchivedManifest copy() {
+    return new HoodieArchivedManifest(new ArrayList<>(this.files));
+  }
+
+  // -------------------------------------------------------------------------
+  //  Inner Class
+  // -------------------------------------------------------------------------
+
+  /**
+   * A file entry.
+   */
+  public static class FileEntry implements Serializable, Comparable<FileEntry> {

Review Comment:
   do we need `numEntries` that tracks the number of timeline entries in each file?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());

Review Comment:
   change log statement to "writing ..`"



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;

Review Comment:
   we need a new HoodieTimelineConfig and pass just that in?



##########
hudi-common/src/main/avro/HoodieArchivedInstant.avsc:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "type":"record",
+   "name":"HoodieArchivedInstant",

Review Comment:
   rename: HoodieLSMTimelineInstant



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));

Review Comment:
   move the `HoodieArchivedTimeline.FILE_LAYER_ZERO` constants here.. the HoodieArchivedTimeline class should be free of any LSM details.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);

Review Comment:
   marker deletion should happen outside of this class, lets add a callback?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();

Review Comment:
   fix configs.. such that there is no trace of "Archive" here in this class.. This way we don't have to rename eventually make a single global timeline. 



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {
+      // 2. sort files by min instant time (implies ascending chronological order)
+      files.sort(HoodieArchivedManifest.FileEntry::compareTo);
+      List<String> candidateFiles = getCandidateFiles(files, compactionBatchSize);
+      if (candidateFiles.size() < 2) {
+        // the file is too large to compact, returns early.
+        return Option.empty();
+      }
+      String compactedFileName = compactedFileName(candidateFiles);
+
+      // 3. compaction
+      compactArchiveFiles(candidateFiles, compactedFileName);
+      // 4. update the manifest file
+      updateManifest(candidateFiles, compactedFileName);
+      LOG.info("Finishes compaction of archive files: " + candidateFiles);
+      return Option.of(compactedFileName);
+    }
+    return Option.empty();
+  }
+
+  public void compactArchiveFiles(List<String> candidateFiles, String compactedFileName) {
+    LOG.info("Starting to merge small archive files.");
+    try (HoodieFileWriter writer = openWriter(new Path(metaClient.getArchivePath(), compactedFileName))) {
+      for (String fileName : candidateFiles) {
+        // Read the archived file
+        try (HoodieAvroParquetReader reader = (HoodieAvroParquetReader) HoodieFileReaderFactory.getReaderFactory(HoodieRecord.HoodieRecordType.AVRO)
+            .getFileReader(metaClient.getHadoopConf(), new Path(metaClient.getArchivePath(), fileName))) {
+          // Read the meta entry
+          try (ClosableIterator<IndexedRecord> iterator = reader.getIndexedRecordIterator(HoodieArchivedInstant.getClassSchema(), HoodieArchivedInstant.getClassSchema())) {
+            while (iterator.hasNext()) {
+              IndexedRecord record = iterator.next();
+              writer.write(record.get(0).toString(), new HoodieAvroIndexedRecord(record), HoodieArchivedInstant.getClassSchema());
+            }
+          }
+        }
+      }
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to merge small archive files", e);
+    }
+    LOG.info("Success to merge small archive files.");
+  }
+
+  /**
+   * Checks whether there is any unfinished compaction operation.
+   *
+   * @param context HoodieEngineContext used for parallelize to delete small archive files if necessary.
+   */
+  public void clean(HoodieEngineContext context, int compactedVersions) throws IOException {
+    // if there are more than 3 version of snapshots, clean the oldest files.

Review Comment:
   how do we cleanup the partially written files, (i.e like the marker mechanism we have) 
   
   - we wrote files (write or compaction) but did not produce manifest. 
   - we wrote a manifest, but crashed before updating version. .. 
   
   My take is to add a new method, that can scan the entire folder and delete any file that is not pointed to by active manifest/versions. We can trigger such cleanup once a day or sth? (or just leave it as a CLI command and people can do it if they wish?)
   



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.

Review Comment:
   when we do this, its important to remember that the next layer may have few files rewritten as well?  do we handle that?
   
   Class LSM ; L0 files may have overlap in ranges, but L1 onward don't. So a single compaction in one level can ripple through to lower levels as well. 
   
   What does the current code do?
   



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {

Review Comment:
   please add a LSMTimeline class that hands out `.reader()` and `.writer()` or rename this class to LSMTimeline and consolidate all read/write methods here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287878544


##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieArchivedManifest.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hudi.common.util.JsonUtils;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Manifest entry for a version snapshot of the archived timeline.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class HoodieArchivedManifest implements Serializable {
+  private static final Logger LOG = LoggerFactory.getLogger(HoodieArchivedManifest.class);
+
+  public static final HoodieArchivedManifest EMPTY = new HoodieArchivedManifest();
+
+  private final List<FileEntry> files;
+
+  // for ser/deser
+  public HoodieArchivedManifest() {
+    this.files = new ArrayList<>();
+  }
+
+  public HoodieArchivedManifest(List<FileEntry> files) {
+    this.files = files;
+  }
+
+  public void addFile(String fileName, long fileLen) {
+    this.files.add(FileEntry.getInstance(fileName, fileLen));
+  }
+
+  public void addFile(FileEntry fileEntry) {
+    this.files.add(fileEntry);
+  }
+
+  public List<FileEntry> getFiles() {
+    return files;
+  }
+
+  public List<String> getFileNames() {
+    return files.stream().map(FileEntry::getFileName).collect(Collectors.toList());
+  }
+
+  // -------------------------------------------------------------------------
+  //  Utilities
+  // -------------------------------------------------------------------------
+
+  public String toJsonString() throws IOException {
+    return JsonUtils.getObjectMapper().writerWithDefaultPrettyPrinter().writeValueAsString(this);
+  }
+
+  public static <T> T fromJsonString(String jsonStr, Class<T> clazz) throws Exception {
+    if (jsonStr == null || jsonStr.isEmpty()) {
+      // For empty commit file (no data or something bad happen).
+      return clazz.newInstance();
+    }
+    return JsonUtils.getObjectMapper().readValue(jsonStr, clazz);
+  }
+
+  public HoodieArchivedManifest copy(List<String> filesToRemove) {
+    List<FileEntry> newFiles = this.files.stream().filter(fileEntry -> !filesToRemove.contains(fileEntry.getFileName())).collect(Collectors.toList());
+    return new HoodieArchivedManifest(newFiles);
+  }
+
+  public HoodieArchivedManifest copy() {
+    return new HoodieArchivedManifest(new ArrayList<>(this.files));
+  }
+
+  // -------------------------------------------------------------------------
+  //  Inner Class
+  // -------------------------------------------------------------------------
+
+  /**
+   * A file entry.
+   */
+  public static class FileEntry implements Serializable, Comparable<FileEntry> {

Review Comment:
   I don't see any request for that now, we can extend it easily in the future because it is a JSON.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287895034


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  private final Map<String, byte[]> readCommits;

Review Comment:
   I just move the old archived timeline codes here, so I prefer not to do any renaming and keep it as it is. The old archived timeline reader loads the timeline by avro lock blocks and yet, it costs a lot of memory.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1696101255

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d688280b7fde224d75843893290db559bcc169f5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d688280b7fde224d75843893290db559bcc169f5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d688280b7fde224d75843893290db559bcc169f5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1305303946


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/LSMTimeline.java:
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * Represents the LSM Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *     \              /
+ *        \        /
+ *            |
+ *            V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *                                      |
+ *                                      V
+ *                                  _version_
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a compacted file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>
+ * Keeps only 3 valid snapshot versions for the reader, that means, a file is kept for at lest 3 archival trigger interval, for default configuration, it is 30 instants time span,
+ * which is far longer that the archived timeline loading time.
+ *
+ * <p><h3>Instants TTL</h3></p>
+ * The timeline reader only reads instants of last limited days. We will by default skip the instants from archived timeline that are generated long time ago.
+ */
+public class LSMTimeline {
+  private static final Logger LOG = LoggerFactory.getLogger(LSMTimeline.class);
+
+  private static final String VERSION_FILE_NAME = "_version_";    // _version_
+  private static final String MANIFEST_FILE_PREFIX = "manifest_"; // manifest_[N]
+
+  private static final String TEMP_FILE_SUFFIX = ".tmp";
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^(\\d+)_(\\d+)_(\\d)\\.parquet");
+
+  // -------------------------------------------------------------------------
+  //  Utilities
+  // -------------------------------------------------------------------------
+  public static Schema getReadSchema(HoodieArchivedTimeline.LoadMode loadMode) {
+    switch (loadMode) {
+      case SLIM:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_SLIM_READ_SCHEMA;
+      case METADATA:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_READ_SCHEMA_WITH_METADATA;
+      case PLAN:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_READ_SCHEMA_WITH_PLAN;
+      default:
+        throw new AssertionError("Unexpected");
+    }
+  }
+
+  /**
+   * Returns whether the given file is located in the filter.
+   */
+  public static boolean isFileInRange(HoodieArchivedTimeline.TimeRangeFilter filter, String fileName) {
+    String minInstant = getMinInstantTime(fileName);
+    String maxInstant = getMaxInstantTime(fileName);
+    return filter.isInRange(minInstant) || filter.isInRange(maxInstant);
+  }
+
+  /**
+   * Returns the latest snapshot version.
+   */
+  public static int latestSnapshotVersion(HoodieTableMetaClient metaClient) throws IOException {
+    Path versionFilePath = getVersionFilePath(metaClient);
+    if (metaClient.getFs().exists(versionFilePath)) {
+      try {
+        Option<byte[]> content = FileIOUtils.readDataFromPath(metaClient.getFs(), versionFilePath);
+        if (content.isPresent()) {
+          return Integer.parseInt(new String(content.get(), StandardCharsets.UTF_8));
+        }
+      } catch (Exception e) {
+        // fallback to manifest file listing.
+        LOG.warn("Error reading version file {}", versionFilePath, e);
+      }
+    }
+    return allSnapshotVersions(metaClient).stream().max(Integer::compareTo).orElse(-1);
+  }
+
+  /**
+   * Returns all the valid snapshot versions.
+   */
+  public static List<Integer> allSnapshotVersions(HoodieTableMetaClient metaClient) throws IOException {
+    return Arrays.stream(metaClient.getFs().listStatus(new Path(metaClient.getArchivePath()), getManifestFilePathFilter()))
+        .map(fileStatus -> fileStatus.getPath().getName())
+        .map(LSMTimeline::getManifestVersion)
+        .collect(Collectors.toList());
+  }
+
+  /**
+   * Returns the latest snapshot metadata files.
+   */
+  public static HoodieLSMTimelineManifest latestSnapshotManifest(HoodieTableMetaClient metaClient) throws IOException {
+    int latestVersion = latestSnapshotVersion(metaClient);
+    return latestSnapshotManifest(metaClient, latestVersion);
+  }
+
+  /**
+   * Reads the file list from the manifest file for the latest snapshot.
+   */
+  public static HoodieLSMTimelineManifest latestSnapshotManifest(HoodieTableMetaClient metaClient, int latestVersion) {
+    if (latestVersion < 0) {
+      // there is no valid snapshot of the timeline.
+      return HoodieLSMTimelineManifest.EMPTY;
+    }
+    // read and deserialize the valid files.
+    byte[] content = FileIOUtils.readDataFromPath(metaClient.getFs(), getManifestFilePath(metaClient, latestVersion)).get();
+    try {
+      return HoodieLSMTimelineManifest.fromJsonString(new String(content, StandardCharsets.UTF_8), HoodieLSMTimelineManifest.class);
+    } catch (Exception e) {
+      throw new HoodieException("Error deserializing manifest entries", e);
+    }
+  }
+
+  /**
+   * Returns the full manifest file path with given version number.
+   */
+  public static Path getManifestFilePath(HoodieTableMetaClient metaClient, int snapshotVersion) {
+    return new Path(metaClient.getArchivePath(), MANIFEST_FILE_PREFIX + snapshotVersion);
+  }
+
+  /**
+   * Returns the full version file path with given version number.
+   */
+  public static Path getVersionFilePath(HoodieTableMetaClient metaClient) {
+    return new Path(metaClient.getArchivePath(), VERSION_FILE_NAME);
+  }
+
+  /**
+   * List all the parquet manifest files.
+   */
+  public static FileStatus[] listAllManifestFiles(HoodieTableMetaClient metaClient) throws IOException {
+    return metaClient.getFs().listStatus(new Path(metaClient.getArchivePath()), getManifestFilePathFilter());
+  }
+
+  /**
+   * List all the parquet metadata files.
+   */
+  public static FileStatus[] listAllMetaFiles(HoodieTableMetaClient metaClient) throws IOException {
+    return metaClient.getFs().globStatus(
+        new Path(metaClient.getArchivePath() + "/*.parquet"));
+  }
+
+  /**
+   * Parse the snapshot version from the manifest file name.
+   */
+  public static int getManifestVersion(String fileName) {
+    return Integer.parseInt(fileName.split("_")[1]);
+  }
+
+  /**
+   * Parse the layer number from the file name.
+   */
+  public static int getFileLayer(String fileName) {
+    try {
+      Matcher fileMatcher = ARCHIVE_FILE_PATTERN.matcher(fileName);
+      if (fileMatcher.matches()) {
+        return Integer.parseInt(fileMatcher.group(3));
+      }
+    } catch (NumberFormatException e) {
+      // log and ignore any format warnings
+      LOG.warn("error getting file layout for archived file: " + fileName);
+    }
+
+    // return default value in case of any errors
+    return 0;
+  }
+
+  /**
+   * Parse the minimum instant time from the file name.
+   */
+  public static String getMinInstantTime(String fileName) {

Review Comment:
   Did see much gains here, we can extend it in the near future if we have more complex name parsing on the file names.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1649620034

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * a7f8558aaffdaab4850780224e1385c3e682372a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1659787826

   > can you please split the code earlier into a "org.apache.hudi.storage.lsm" package? where we keep the parquet LSM code away from its use in the ArchivedTimeline? I think it ll help with even reviewing code a lot.
   
   Moved the LSM write path codes into `ArchivedTimelineWriter`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1663993572

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 57c1b843608a9b63d143ead5dd5168613bb13969 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287876922


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  private final Map<String, byte[]> readCommits;
+
+  public LegacyArchivedMetaEntryReader(HoodieTableMetaClient metaClient) {
+    this.metaClient = metaClient;
+    this.readCommits = new HashMap<>();
+  }
+
+  /**
+   * @deprecated Use {@code HoodieArchivedTimeline#readCommit} instead.

Review Comment:
   Should be fine~



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287896192


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
+import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.function.Function;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.StreamSupport;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  private final Map<String, byte[]> readCommits;
+
+  public LegacyArchivedMetaEntryReader(HoodieTableMetaClient metaClient) {
+    this.metaClient = metaClient;
+    this.readCommits = new HashMap<>();
+  }
+
+  /**
+   * @deprecated Use {@code HoodieArchivedTimeline#readCommit} instead.
+   */
+  private HoodieInstant readCommit(GenericRecord record, boolean loadDetails) {
+    final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
+    final String action = record.get(ACTION_TYPE_KEY).toString();
+    final String stateTransitionTime = (String) record.get(STATE_TRANSITION_TIME);
+    if (loadDetails) {
+      getMetadataKey(action).map(key -> {
+        Object actionData = record.get(key);
+        if (actionData != null) {
+          if (action.equals(HoodieTimeline.COMPACTION_ACTION)) {
+            this.readCommits.put(instantTime, HoodieAvroUtils.indexedRecordToBytes((IndexedRecord) actionData));
+          } else {
+            this.readCommits.put(instantTime, actionData.toString().getBytes(StandardCharsets.UTF_8));
+          }
+        }
+        return null;
+      });
+    }
+    return new HoodieInstant(HoodieInstant.State.valueOf(record.get(ACTION_STATE).toString()), action,
+        instantTime, stateTransitionTime);
+  }
+
+  @Nonnull
+  private Option<String> getMetadataKey(String action) {
+    switch (action) {
+      case HoodieTimeline.CLEAN_ACTION:
+        return Option.of("hoodieCleanMetadata");
+      case HoodieTimeline.COMMIT_ACTION:
+      case HoodieTimeline.DELTA_COMMIT_ACTION:
+        return Option.of("hoodieCommitMetadata");
+      case HoodieTimeline.ROLLBACK_ACTION:
+        return Option.of("hoodieRollbackMetadata");
+      case HoodieTimeline.SAVEPOINT_ACTION:
+        return Option.of("hoodieSavePointMetadata");
+      case HoodieTimeline.COMPACTION_ACTION:
+      case HoodieTimeline.LOG_COMPACTION_ACTION:
+        return Option.of("hoodieCompactionPlan");
+      case HoodieTimeline.REPLACE_COMMIT_ACTION:
+        return Option.of("hoodieReplaceCommitMetadata");
+      case HoodieTimeline.INDEXING_ACTION:
+        return Option.of("hoodieIndexCommitMetadata");
+      default:
+        LOG.error(String.format("Unknown action in metadata (%s)", action));
+        return Option.empty();
+    }
+  }
+
+  /**
+   * This is method to read selected instants. Do NOT use this directly use one of the helper methods above
+   * If loadInstantDetails is set to true, this would also update 'readCommits' map with commit details
+   * If filter is specified, only the filtered instants are loaded
+   * If commitsFilter is specified, only the filtered records are loaded
+   *
+   * @deprecated Use {@code HoodieArchivedTimeline.loadInstantsV2} instead.

Review Comment:
   The comment is overdue, we can remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288170683


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {
+      // 2. sort files by min instant time (implies ascending chronological order)
+      files.sort(HoodieArchivedManifest.FileEntry::compareTo);
+      List<String> candidateFiles = getCandidateFiles(files, compactionBatchSize);
+      if (candidateFiles.size() < 2) {
+        // the file is too large to compact, returns early.
+        return Option.empty();
+      }
+      String compactedFileName = compactedFileName(candidateFiles);
+
+      // 3. compaction
+      compactArchiveFiles(candidateFiles, compactedFileName);
+      // 4. update the manifest file
+      updateManifest(candidateFiles, compactedFileName);
+      LOG.info("Finishes compaction of archive files: " + candidateFiles);
+      return Option.of(compactedFileName);
+    }
+    return Option.empty();
+  }
+
+  public void compactArchiveFiles(List<String> candidateFiles, String compactedFileName) {
+    LOG.info("Starting to merge small archive files.");
+    try (HoodieFileWriter writer = openWriter(new Path(metaClient.getArchivePath(), compactedFileName))) {
+      for (String fileName : candidateFiles) {
+        // Read the archived file
+        try (HoodieAvroParquetReader reader = (HoodieAvroParquetReader) HoodieFileReaderFactory.getReaderFactory(HoodieRecord.HoodieRecordType.AVRO)
+            .getFileReader(metaClient.getHadoopConf(), new Path(metaClient.getArchivePath(), fileName))) {
+          // Read the meta entry
+          try (ClosableIterator<IndexedRecord> iterator = reader.getIndexedRecordIterator(HoodieArchivedInstant.getClassSchema(), HoodieArchivedInstant.getClassSchema())) {
+            while (iterator.hasNext()) {
+              IndexedRecord record = iterator.next();
+              writer.write(record.get(0).toString(), new HoodieAvroIndexedRecord(record), HoodieArchivedInstant.getClassSchema());
+            }
+          }
+        }
+      }
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to merge small archive files", e);
+    }
+    LOG.info("Success to merge small archive files.");
+  }
+
+  /**
+   * Checks whether there is any unfinished compaction operation.
+   *
+   * @param context HoodieEngineContext used for parallelize to delete small archive files if necessary.
+   */
+  public void clean(HoodieEngineContext context, int compactedVersions) throws IOException {
+    // if there are more than 3 version of snapshots, clean the oldest files.

Review Comment:
   Is this clean, we already list all the data files first, and diff that with the valid version files in our latest snapshots, all those invalid/partial written files should be removed.
   
   We clean the manifest file in the similiar way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1692927735

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 803df61d0d04f7e7403d1177325a365e9bbafab5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226) 
   * 2288afba514c322841e5479a7fc7f6cdc8bf8b2d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1661438646

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 9a2afc24ea8552ec6869f9d5819a00c78067a4e6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955) 
   * 6ab7d4bd2696988e3784291879d1f713d6a9c98d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6539] New LSM tree style archived timeline [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1393515340


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -164,230 +101,32 @@ public boolean archiveIfRequired(HoodieEngineContext context, boolean acquireLoc
         // there is no owner or instant time per se for archival.
         txnManager.beginTransaction(Option.empty(), Option.empty());
       }
-      List<HoodieInstant> instantsToArchive = getInstantsToArchive().collect(Collectors.toList());
-      verifyLastMergeArchiveFilesIfNecessary(context);
+      // Sort again because the cleaning and rollback instants could break the sequence.
+      List<ActiveAction> instantsToArchive = getInstantsToArchive().sorted().collect(Collectors.toList());
       boolean success = true;
       if (!instantsToArchive.isEmpty()) {
-        this.writer = openWriter();
         LOG.info("Archiving instants " + instantsToArchive);
-        archive(context, instantsToArchive);
+        Consumer<Exception> exceptionHandler = e -> {
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw new HoodieException(e);
+          }
+        };
+        this.timelineWriter.write(instantsToArchive, Option.of(action -> deleteAnyLeftOverMarkers(context, action)), Option.of(exceptionHandler));
         LOG.info("Deleting archived instants " + instantsToArchive);
         success = deleteArchivedInstants(instantsToArchive, context);

Review Comment:
   we should, but currently the redundants left by failed deletion does not affect the correctness, the completion time would be still loaded correctly if an instant locates both avtive and archived timeline.
   
   We need to thing through the design though, it is arduous to maintain the whole multiple handling as atomic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1300866432


##########
hudi-common/src/main/avro/HoodieLSMTimelineInstant.avsc:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "type":"record",
+   "name":"HoodieLSMTimelineInstant",
+   "namespace":"org.apache.hudi.avro.model",
+   "fields":[
+      {
+         "name":"instantTime",
+         "type":["null","string"],
+         "default": null
+      },
+      {
+         "name":"completionTime",
+         "type":["null","string"],
+         "default": null
+      },
+      {
+         "name":"action",
+         "type":["null","string"],
+         "default": null
+      },
+      {
+         "name":"metadata",
+         "type":["null", "bytes"],
+         "default": null
+      },
+      {
+         "name":"plan",
+         "type":["null", "bytes"],

Review Comment:
   I am thinking of the scenario where we want users to write SQL to query the timeline. if we do bytes, we need to probably provide udfs for converting from bytes to a plan? Follow up JIRA? (I think this is still better than nested schema, which can be expensive to write.)



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveAction.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;

Review Comment:
   can we make a org.apache.hudi.client.timeline package and move all these classes there. 



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LSMTimelineWriter.java:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieLSMTimelineInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.LSMTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class LSMTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(LSMTimelineWriter.class);
+
+  public static final int FILE_LAYER_ZERO = 0;
+
+  public static final long MAX_FILE_SIZE_IN_BYTES = 1024 * 1024 * 1000;
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private LSMTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static LSMTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new LSMTimelineWriter(config, table);
+  }
+
+  public void write(List<ActiveAction> activeActions, Consumer<ActiveAction> preWriteCallback) throws HoodieCommitException {
+    ValidationUtils.checkArgument(!activeActions.isEmpty(), "The instant actions to write should not be empty");
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(activeActions.get(0).getInstantTime(), activeActions.get(activeActions.size() - 1).getInstantTime(), FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieLSMTimelineInstant.getClassSchema();
+      LOG.info("Writing schema " + wrapperSchema.toString());
+      for (ActiveAction activeAction : activeActions) {
+        try {
+          if (preWriteCallback != null) {
+            preWriteCallback.accept(activeAction);
+          }
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieLSMTimelineInstant metaEntry = MetadataConversionUtils.createArchivedInstant(activeAction, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to write instant: " + activeAction.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {

Review Comment:
   move this handling out of this class?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LegacyArchivedMetaEntryReader.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Tools used for migrating to new LSM tree style archived timeline.
+ */
+public class LegacyArchivedMetaEntryReader {
+  private static final Logger LOG = LoggerFactory.getLogger(LegacyArchivedMetaEntryReader.class);
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^\\.commits_\\.archive\\.([0-9]+).*");
+
+  public static final String MERGE_ARCHIVE_PLAN_NAME = "mergeArchivePlan";
+
+  private static final String ACTION_TYPE_KEY = "actionType";
+  private static final String ACTION_STATE = "actionState";
+  private static final String STATE_TRANSITION_TIME = "stateTransitionTime";
+
+  private final HoodieTableMetaClient metaClient;
+
+  public LegacyArchivedMetaEntryReader(HoodieTableMetaClient metaClient) {
+    this.metaClient = metaClient;
+  }
+
+  public ClosableIterator<ActiveAction> getActiveActionsIterator() {
+    return loadInstants(null);
+  }
+
+  /**
+   * Reads the avro record for instant and details.
+   */
+  private Pair<HoodieInstant, Option<byte[]>> readInstant(GenericRecord record) {
+    final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
+    final String action = record.get(ACTION_TYPE_KEY).toString();
+    final String stateTransitionTime = (String) record.get(STATE_TRANSITION_TIME);
+    final Option<byte[]> details = getMetadataKey(action).map(key -> {
+      Object actionData = record.get(key);
+      if (actionData != null) {
+        if (action.equals(HoodieTimeline.COMPACTION_ACTION)) {
+          return HoodieAvroUtils.indexedRecordToBytes((IndexedRecord) actionData);
+        } else {
+          return actionData.toString().getBytes(StandardCharsets.UTF_8);
+        }
+      }
+      return null;
+    });
+    HoodieInstant instant = new HoodieInstant(HoodieInstant.State.valueOf(record.get(ACTION_STATE).toString()), action,
+        instantTime, stateTransitionTime);
+    return Pair.of(instant,details);
+  }
+
+  @Nonnull
+  private Option<String> getMetadataKey(String action) {
+    switch (action) {
+      case HoodieTimeline.CLEAN_ACTION:
+        return Option.of("hoodieCleanMetadata");

Review Comment:
   any existing constants we can use for this? else ignore.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LSMTimelineWriter.java:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieLSMTimelineInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.LSMTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class LSMTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(LSMTimelineWriter.class);
+
+  public static final int FILE_LAYER_ZERO = 0;
+
+  public static final long MAX_FILE_SIZE_IN_BYTES = 1024 * 1024 * 1000;
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private LSMTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static LSMTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new LSMTimelineWriter(config, table);
+  }
+
+  public void write(List<ActiveAction> activeActions, Consumer<ActiveAction> preWriteCallback) throws HoodieCommitException {
+    ValidationUtils.checkArgument(!activeActions.isEmpty(), "The instant actions to write should not be empty");
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(activeActions.get(0).getInstantTime(), activeActions.get(activeActions.size() - 1).getInstantTime(), FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieLSMTimelineInstant.getClassSchema();
+      LOG.info("Writing schema " + wrapperSchema.toString());
+      for (ActiveAction activeAction : activeActions) {
+        try {
+          if (preWriteCallback != null) {
+            preWriteCallback.accept(activeAction);
+          }
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieLSMTimelineInstant metaEntry = MetadataConversionUtils.createArchivedInstant(activeAction, metaClient);

Review Comment:
   rename. remove any reference to "Archived"



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LSMTimelineWriter.java:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieLSMTimelineInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.LSMTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.

Review Comment:
   remove "archived"



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LSMTimelineWriter.java:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieLSMTimelineInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.LSMTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class LSMTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(LSMTimelineWriter.class);
+
+  public static final int FILE_LAYER_ZERO = 0;
+
+  public static final long MAX_FILE_SIZE_IN_BYTES = 1024 * 1024 * 1000;
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private LSMTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static LSMTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new LSMTimelineWriter(config, table);
+  }
+
+  public void write(List<ActiveAction> activeActions, Consumer<ActiveAction> preWriteCallback) throws HoodieCommitException {
+    ValidationUtils.checkArgument(!activeActions.isEmpty(), "The instant actions to write should not be empty");
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(activeActions.get(0).getInstantTime(), activeActions.get(activeActions.size() - 1).getInstantTime(), FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieLSMTimelineInstant.getClassSchema();
+      LOG.info("Writing schema " + wrapperSchema.toString());
+      for (ActiveAction activeAction : activeActions) {
+        try {
+          if (preWriteCallback != null) {

Review Comment:
   can we try to use Option instead of `null` as sentinels?



##########
hudi-common/src/main/avro/HoodieLSMTimelineInstant.avsc:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "type":"record",
+   "name":"HoodieLSMTimelineInstant",
+   "namespace":"org.apache.hudi.avro.model",
+   "fields":[

Review Comment:
   lets add a version field to each record, so we can evolve as we go if needed



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LSMTimelineWriter.java:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieLSMTimelineInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.LSMTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class LSMTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(LSMTimelineWriter.class);
+
+  public static final int FILE_LAYER_ZERO = 0;
+
+  public static final long MAX_FILE_SIZE_IN_BYTES = 1024 * 1024 * 1000;
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private LSMTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static LSMTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new LSMTimelineWriter(config, table);
+  }
+
+  public void write(List<ActiveAction> activeActions, Consumer<ActiveAction> preWriteCallback) throws HoodieCommitException {
+    ValidationUtils.checkArgument(!activeActions.isEmpty(), "The instant actions to write should not be empty");
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(activeActions.get(0).getInstantTime(), activeActions.get(activeActions.size() - 1).getInstantTime(), FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieLSMTimelineInstant.getClassSchema();
+      LOG.info("Writing schema " + wrapperSchema.toString());
+      for (ActiveAction activeAction : activeActions) {
+        try {
+          if (preWriteCallback != null) {
+            preWriteCallback.accept(activeAction);
+          }
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieLSMTimelineInstant metaEntry = MetadataConversionUtils.createArchivedInstant(activeAction, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to write instant: " + activeAction.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to write commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = LSMTimeline.latestSnapshotVersion(metaClient);
+    HoodieLSMTimelineManifest latestManifest = LSMTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieLSMTimelineManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;

Review Comment:
   multi line comment or move to java doc?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LSMTimelineWriter.java:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieLSMTimelineInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.LSMTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class LSMTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(LSMTimelineWriter.class);
+
+  public static final int FILE_LAYER_ZERO = 0;
+
+  public static final long MAX_FILE_SIZE_IN_BYTES = 1024 * 1024 * 1000;
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private LSMTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static LSMTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new LSMTimelineWriter(config, table);
+  }
+
+  public void write(List<ActiveAction> activeActions, Consumer<ActiveAction> preWriteCallback) throws HoodieCommitException {
+    ValidationUtils.checkArgument(!activeActions.isEmpty(), "The instant actions to write should not be empty");
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(activeActions.get(0).getInstantTime(), activeActions.get(activeActions.size() - 1).getInstantTime(), FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieLSMTimelineInstant.getClassSchema();
+      LOG.info("Writing schema " + wrapperSchema.toString());
+      for (ActiveAction activeAction : activeActions) {
+        try {
+          if (preWriteCallback != null) {
+            preWriteCallback.accept(activeAction);
+          }
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieLSMTimelineInstant metaEntry = MetadataConversionUtils.createArchivedInstant(activeAction, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to write instant: " + activeAction.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to write commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = LSMTimeline.latestSnapshotVersion(metaClient);
+    HoodieLSMTimelineManifest latestManifest = LSMTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieLSMTimelineManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = LSMTimeline.latestSnapshotVersion(metaClient);
+    HoodieLSMTimelineManifest latestManifest = LSMTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieLSMTimelineManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieLSMTimelineManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = LSMTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = LSMTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.

Review Comment:
   please look for all references of "archive" and cleanup.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java:
##########
@@ -424,9 +159,7 @@ private Stream<HoodieInstant> getCommitInstantsToArchive() throws IOException {
                   LESSER_THAN, oldestPendingInstant.get().getTimestamp())).findFirst());
       // Check if the completed instant is higher than the oldest inflight instant
       // in that case update the oldestCommitToRetain to oldestInflight commit time.
-      if (!completedCommitBeforeOldestPendingInstant.isPresent()
-          || HoodieTimeline.compareTimestamps(oldestPendingInstant.get().getTimestamp(),
-          LESSER_THAN, completedCommitBeforeOldestPendingInstant.get().getTimestamp())) {
+      if (!completedCommitBeforeOldestPendingInstant.isPresent()) {

Review Comment:
   why do we need this change? this may break sth. please revert if unnecessary for this change. 



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/LSMTimeline.java:
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * Represents the LSM Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *     \              /
+ *        \        /
+ *            |
+ *            V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *                                      |
+ *                                      V
+ *                                  _version_
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a compacted file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>
+ * Keeps only 3 valid snapshot versions for the reader, that means, a file is kept for at lest 3 archival trigger interval, for default configuration, it is 30 instants time span,
+ * which is far longer that the archived timeline loading time.
+ *
+ * <p><h3>Instants TTL</h3></p>
+ * The timeline reader only reads instants of last limited days. We will by default skip the instants from archived timeline that are generated long time ago.
+ */
+public class LSMTimeline {
+  private static final Logger LOG = LoggerFactory.getLogger(LSMTimeline.class);
+
+  private static final String VERSION_FILE_NAME = "_version_";    // _version_
+  private static final String MANIFEST_FILE_PREFIX = "manifest_"; // manifest_[N]
+
+  private static final String TEMP_FILE_SUFFIX = ".tmp";
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^(\\d+)_(\\d+)_(\\d)\\.parquet");
+
+  // -------------------------------------------------------------------------
+  //  Utilities
+  // -------------------------------------------------------------------------
+  public static Schema getReadSchema(HoodieArchivedTimeline.LoadMode loadMode) {
+    switch (loadMode) {
+      case SLIM:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_SLIM_READ_SCHEMA;
+      case METADATA:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_READ_SCHEMA_WITH_METADATA;
+      case PLAN:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_READ_SCHEMA_WITH_PLAN;
+      default:
+        throw new AssertionError("Unexpected");
+    }
+  }
+
+  /**
+   * Returns whether the given file is located in the filter.
+   */
+  public static boolean isFileInRange(HoodieArchivedTimeline.TimeRangeFilter filter, String fileName) {
+    String minInstant = getMinInstantTime(fileName);
+    String maxInstant = getMaxInstantTime(fileName);
+    return filter.isInRange(minInstant) || filter.isInRange(maxInstant);
+  }
+
+  /**
+   * Returns the latest snapshot version.
+   */
+  public static int latestSnapshotVersion(HoodieTableMetaClient metaClient) throws IOException {
+    Path versionFilePath = getVersionFilePath(metaClient);
+    if (metaClient.getFs().exists(versionFilePath)) {
+      try {
+        Option<byte[]> content = FileIOUtils.readDataFromPath(metaClient.getFs(), versionFilePath);
+        if (content.isPresent()) {
+          return Integer.parseInt(new String(content.get(), StandardCharsets.UTF_8));
+        }
+      } catch (Exception e) {
+        // fallback to manifest file listing.
+        LOG.warn("Error reading version file {}", versionFilePath, e);
+      }
+    }
+    return allSnapshotVersions(metaClient).stream().max(Integer::compareTo).orElse(-1);
+  }
+
+  /**
+   * Returns all the valid snapshot versions.
+   */
+  public static List<Integer> allSnapshotVersions(HoodieTableMetaClient metaClient) throws IOException {
+    return Arrays.stream(metaClient.getFs().listStatus(new Path(metaClient.getArchivePath()), getManifestFilePathFilter()))
+        .map(fileStatus -> fileStatus.getPath().getName())
+        .map(LSMTimeline::getManifestVersion)
+        .collect(Collectors.toList());
+  }
+
+  /**
+   * Returns the latest snapshot metadata files.
+   */
+  public static HoodieLSMTimelineManifest latestSnapshotManifest(HoodieTableMetaClient metaClient) throws IOException {
+    int latestVersion = latestSnapshotVersion(metaClient);
+    return latestSnapshotManifest(metaClient, latestVersion);
+  }
+
+  /**
+   * Reads the file list from the manifest file for the latest snapshot.
+   */
+  public static HoodieLSMTimelineManifest latestSnapshotManifest(HoodieTableMetaClient metaClient, int latestVersion) {
+    if (latestVersion < 0) {
+      // there is no valid snapshot of the timeline.
+      return HoodieLSMTimelineManifest.EMPTY;
+    }
+    // read and deserialize the valid files.
+    byte[] content = FileIOUtils.readDataFromPath(metaClient.getFs(), getManifestFilePath(metaClient, latestVersion)).get();
+    try {
+      return HoodieLSMTimelineManifest.fromJsonString(new String(content, StandardCharsets.UTF_8), HoodieLSMTimelineManifest.class);
+    } catch (Exception e) {
+      throw new HoodieException("Error deserializing manifest entries", e);
+    }
+  }
+
+  /**
+   * Returns the full manifest file path with given version number.
+   */
+  public static Path getManifestFilePath(HoodieTableMetaClient metaClient, int snapshotVersion) {
+    return new Path(metaClient.getArchivePath(), MANIFEST_FILE_PREFIX + snapshotVersion);
+  }
+
+  /**
+   * Returns the full version file path with given version number.
+   */
+  public static Path getVersionFilePath(HoodieTableMetaClient metaClient) {
+    return new Path(metaClient.getArchivePath(), VERSION_FILE_NAME);
+  }
+
+  /**
+   * List all the parquet manifest files.
+   */
+  public static FileStatus[] listAllManifestFiles(HoodieTableMetaClient metaClient) throws IOException {
+    return metaClient.getFs().listStatus(new Path(metaClient.getArchivePath()), getManifestFilePathFilter());
+  }
+
+  /**
+   * List all the parquet metadata files.
+   */
+  public static FileStatus[] listAllMetaFiles(HoodieTableMetaClient metaClient) throws IOException {
+    return metaClient.getFs().globStatus(
+        new Path(metaClient.getArchivePath() + "/*.parquet"));
+  }
+
+  /**
+   * Parse the snapshot version from the manifest file name.
+   */
+  public static int getManifestVersion(String fileName) {
+    return Integer.parseInt(fileName.split("_")[1]);
+  }
+
+  /**
+   * Parse the layer number from the file name.
+   */
+  public static int getFileLayer(String fileName) {
+    try {
+      Matcher fileMatcher = ARCHIVE_FILE_PATTERN.matcher(fileName);
+      if (fileMatcher.matches()) {
+        return Integer.parseInt(fileMatcher.group(3));
+      }
+    } catch (NumberFormatException e) {
+      // log and ignore any format warnings
+      LOG.warn("error getting file layout for archived file: " + fileName);
+    }
+
+    // return default value in case of any errors
+    return 0;
+  }
+
+  /**
+   * Parse the minimum instant time from the file name.
+   */
+  public static String getMinInstantTime(String fileName) {
+    Matcher fileMatcher = ARCHIVE_FILE_PATTERN.matcher(fileName);
+    if (fileMatcher.matches()) {
+      return fileMatcher.group(1);
+    } else {
+      throw new HoodieException("Unexpected archival file name: " + fileName);
+    }
+  }
+
+  /**
+   * Parse the maximum instant time from the file name.
+   */
+  public static String getMaxInstantTime(String fileName) {

Review Comment:
   do these methods have UTs?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LSMTimelineWriter.java:
##########
@@ -0,0 +1,350 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieLSMTimelineInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.LSMTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class LSMTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(LSMTimelineWriter.class);
+
+  public static final int FILE_LAYER_ZERO = 0;
+
+  public static final long MAX_FILE_SIZE_IN_BYTES = 1024 * 1024 * 1000;
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private LSMTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static LSMTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new LSMTimelineWriter(config, table);
+  }
+
+  public void write(List<ActiveAction> activeActions, Consumer<ActiveAction> preWriteCallback) throws HoodieCommitException {
+    ValidationUtils.checkArgument(!activeActions.isEmpty(), "The instant actions to write should not be empty");
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(activeActions.get(0).getInstantTime(), activeActions.get(activeActions.size() - 1).getInstantTime(), FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieLSMTimelineInstant.getClassSchema();
+      LOG.info("Writing schema " + wrapperSchema.toString());
+      for (ActiveAction activeAction : activeActions) {
+        try {
+          if (preWriteCallback != null) {
+            preWriteCallback.accept(activeAction);
+          }
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieLSMTimelineInstant metaEntry = MetadataConversionUtils.createArchivedInstant(activeAction, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to write instant: " + activeAction.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to write commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = LSMTimeline.latestSnapshotVersion(metaClient);
+    HoodieLSMTimelineManifest latestManifest = LSMTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieLSMTimelineManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = LSMTimeline.latestSnapshotVersion(metaClient);
+    HoodieLSMTimelineManifest latestManifest = LSMTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieLSMTimelineManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieLSMTimelineManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = LSMTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = LSMTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieLSMTimelineManifest latestManifest = LSMTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieLSMTimelineManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieLSMTimelineManifest.LSMFileEntry> files = manifest.getFiles()
+        .stream().filter(file -> LSMTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getTimelineCompactionBatchSize();
+
+    if (files.size() >= compactionBatchSize) {
+      // 2. sort files by min instant time (implies ascending chronological order)
+      files.sort(HoodieLSMTimelineManifest.LSMFileEntry::compareTo);
+      List<String> candidateFiles = getCandidateFiles(files, compactionBatchSize);
+      if (candidateFiles.size() < 2) {
+        // the file is too large to compact, returns early.
+        return Option.empty();
+      }
+      String compactedFileName = compactedFileName(candidateFiles);
+
+      // 3. compaction
+      compactFiles(candidateFiles, compactedFileName);
+      // 4. update the manifest file
+      updateManifest(candidateFiles, compactedFileName);
+      LOG.info("Finishes compaction of archive files: " + candidateFiles);
+      return Option.of(compactedFileName);
+    }
+    return Option.empty();
+  }
+
+  public void compactFiles(List<String> candidateFiles, String compactedFileName) {
+    LOG.info("Starting to merge small archive files.");
+    try (HoodieFileWriter writer = openWriter(new Path(metaClient.getArchivePath(), compactedFileName))) {
+      for (String fileName : candidateFiles) {
+        // Read the archived file
+        try (HoodieAvroParquetReader reader = (HoodieAvroParquetReader) HoodieFileReaderFactory.getReaderFactory(HoodieRecord.HoodieRecordType.AVRO)
+            .getFileReader(metaClient.getHadoopConf(), new Path(metaClient.getArchivePath(), fileName))) {
+          // Read the meta entry
+          try (ClosableIterator<IndexedRecord> iterator = reader.getIndexedRecordIterator(HoodieLSMTimelineInstant.getClassSchema(), HoodieLSMTimelineInstant.getClassSchema())) {
+            while (iterator.hasNext()) {
+              IndexedRecord record = iterator.next();
+              writer.write(record.get(0).toString(), new HoodieAvroIndexedRecord(record), HoodieLSMTimelineInstant.getClassSchema());
+            }
+          }
+        }
+      }
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to merge small archive files", e);
+    }
+    LOG.info("Success to merge small archive files.");
+  }
+
+  /**
+   * Checks whether there is any unfinished compaction operation.
+   *
+   * @param context HoodieEngineContext used for parallelize to delete small archive files if necessary.
+   */
+  public void clean(HoodieEngineContext context, int compactedVersions) throws IOException {
+    // if there are more than 3 version of snapshots, clean the oldest files.
+    List<Integer> allSnapshotVersions = LSMTimeline.allSnapshotVersions(metaClient);
+    int numVersionsToKeep = 3 + compactedVersions; // should make the threshold configurable.
+    if (allSnapshotVersions.size() > numVersionsToKeep) {
+      allSnapshotVersions.sort((v1, v2) -> v2 - v1);
+      List<Integer> versionsToKeep = allSnapshotVersions.subList(0, numVersionsToKeep);
+      Set<String> filesToKeep = versionsToKeep.stream()
+          .flatMap(version -> LSMTimeline.latestSnapshotManifest(metaClient, version).getFileNames().stream())
+          .collect(Collectors.toSet());
+      // delete the manifest file first
+      List<String> manifestFilesToClean = new ArrayList<>();
+      Arrays.stream(LSMTimeline.listAllManifestFiles(metaClient)).forEach(fileStatus -> {
+        if (!versionsToKeep.contains(LSMTimeline.getManifestVersion(fileStatus.getPath().getName()))) {
+          manifestFilesToClean.add(fileStatus.getPath().toString());
+        }
+      });
+      FSUtils.deleteFilesParallelize(metaClient, manifestFilesToClean, context, config.getArchiveDeleteParallelism(), false);
+      // delete the archive data files
+      List<String> dataFilesToClean = Arrays.stream(LSMTimeline.listAllMetaFiles(metaClient))
+          .filter(fileStatus -> !filesToKeep.contains(fileStatus.getPath().getName()))
+          .map(fileStatus -> fileStatus.getPath().toString())
+          .collect(Collectors.toList());
+      FSUtils.deleteFilesParallelize(metaClient, dataFilesToClean, context, config.getArchiveDeleteParallelism(), false);
+    }
+  }
+
+  private HoodieLSMTimelineManifest.LSMFileEntry getFileEntry(String fileName) throws IOException {
+    long fileLen = metaClient.getFs().getFileStatus(new Path(metaClient.getArchivePath(), fileName)).getLen();
+    return HoodieLSMTimelineManifest.LSMFileEntry.getInstance(fileName, fileLen);
+  }
+
+  /**
+   * Returns at most {@code filesBatch} number of source files
+   * restricted by the gross file size by 1GB.
+   */
+  private List<String> getCandidateFiles(List<HoodieLSMTimelineManifest.LSMFileEntry> files, int filesBatch) throws IOException {
+    List<String> candidates = new ArrayList<>();
+    long totalFileLen = 0L;
+    for (int i = 0; i < filesBatch; i++) {
+      HoodieLSMTimelineManifest.LSMFileEntry fileEntry = files.get(i);
+      if (totalFileLen > MAX_FILE_SIZE_IN_BYTES) {
+        return candidates;
+      }
+      // we may also need to consider a single file that is very close to the threshold in size,
+      // to avoid the write amplification,
+      // for e.g, two 800MB files compact into a 1.6GB file.
+      totalFileLen += fileEntry.getFileLen();
+      candidates.add(fileEntry.getFileName());
+    }
+    return candidates;
+  }
+
+  /**
+   * Returns a new file name.
+   */
+  private static String newFileName(String minInstant, String maxInstant, int layer) {
+    return minInstant + "_" + maxInstant + "_" + layer + HoodieFileFormat.PARQUET.getFileExtension();

Review Comment:
   String.format?



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/LSMTimeline.java:
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * Represents the LSM Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *     \              /
+ *        \        /
+ *            |
+ *            V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *                                      |
+ *                                      V
+ *                                  _version_
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a compacted file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>
+ * Keeps only 3 valid snapshot versions for the reader, that means, a file is kept for at lest 3 archival trigger interval, for default configuration, it is 30 instants time span,
+ * which is far longer that the archived timeline loading time.
+ *
+ * <p><h3>Instants TTL</h3></p>
+ * The timeline reader only reads instants of last limited days. We will by default skip the instants from archived timeline that are generated long time ago.
+ */
+public class LSMTimeline {
+  private static final Logger LOG = LoggerFactory.getLogger(LSMTimeline.class);
+
+  private static final String VERSION_FILE_NAME = "_version_";    // _version_
+  private static final String MANIFEST_FILE_PREFIX = "manifest_"; // manifest_[N]
+
+  private static final String TEMP_FILE_SUFFIX = ".tmp";
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^(\\d+)_(\\d+)_(\\d)\\.parquet");
+
+  // -------------------------------------------------------------------------
+  //  Utilities
+  // -------------------------------------------------------------------------
+  public static Schema getReadSchema(HoodieArchivedTimeline.LoadMode loadMode) {
+    switch (loadMode) {
+      case SLIM:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_SLIM_READ_SCHEMA;
+      case METADATA:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_READ_SCHEMA_WITH_METADATA;
+      case PLAN:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_READ_SCHEMA_WITH_PLAN;
+      default:
+        throw new AssertionError("Unexpected");
+    }
+  }
+
+  /**
+   * Returns whether the given file is located in the filter.
+   */
+  public static boolean isFileInRange(HoodieArchivedTimeline.TimeRangeFilter filter, String fileName) {
+    String minInstant = getMinInstantTime(fileName);
+    String maxInstant = getMaxInstantTime(fileName);
+    return filter.isInRange(minInstant) || filter.isInRange(maxInstant);
+  }
+
+  /**
+   * Returns the latest snapshot version.
+   */
+  public static int latestSnapshotVersion(HoodieTableMetaClient metaClient) throws IOException {
+    Path versionFilePath = getVersionFilePath(metaClient);
+    if (metaClient.getFs().exists(versionFilePath)) {
+      try {
+        Option<byte[]> content = FileIOUtils.readDataFromPath(metaClient.getFs(), versionFilePath);
+        if (content.isPresent()) {
+          return Integer.parseInt(new String(content.get(), StandardCharsets.UTF_8));
+        }
+      } catch (Exception e) {
+        // fallback to manifest file listing.
+        LOG.warn("Error reading version file {}", versionFilePath, e);
+      }
+    }
+    return allSnapshotVersions(metaClient).stream().max(Integer::compareTo).orElse(-1);
+  }
+
+  /**
+   * Returns all the valid snapshot versions.
+   */
+  public static List<Integer> allSnapshotVersions(HoodieTableMetaClient metaClient) throws IOException {
+    return Arrays.stream(metaClient.getFs().listStatus(new Path(metaClient.getArchivePath()), getManifestFilePathFilter()))
+        .map(fileStatus -> fileStatus.getPath().getName())
+        .map(LSMTimeline::getManifestVersion)
+        .collect(Collectors.toList());
+  }
+
+  /**
+   * Returns the latest snapshot metadata files.
+   */
+  public static HoodieLSMTimelineManifest latestSnapshotManifest(HoodieTableMetaClient metaClient) throws IOException {
+    int latestVersion = latestSnapshotVersion(metaClient);
+    return latestSnapshotManifest(metaClient, latestVersion);
+  }
+
+  /**
+   * Reads the file list from the manifest file for the latest snapshot.
+   */
+  public static HoodieLSMTimelineManifest latestSnapshotManifest(HoodieTableMetaClient metaClient, int latestVersion) {
+    if (latestVersion < 0) {
+      // there is no valid snapshot of the timeline.
+      return HoodieLSMTimelineManifest.EMPTY;
+    }
+    // read and deserialize the valid files.
+    byte[] content = FileIOUtils.readDataFromPath(metaClient.getFs(), getManifestFilePath(metaClient, latestVersion)).get();
+    try {
+      return HoodieLSMTimelineManifest.fromJsonString(new String(content, StandardCharsets.UTF_8), HoodieLSMTimelineManifest.class);
+    } catch (Exception e) {
+      throw new HoodieException("Error deserializing manifest entries", e);
+    }
+  }
+
+  /**
+   * Returns the full manifest file path with given version number.
+   */
+  public static Path getManifestFilePath(HoodieTableMetaClient metaClient, int snapshotVersion) {
+    return new Path(metaClient.getArchivePath(), MANIFEST_FILE_PREFIX + snapshotVersion);
+  }
+
+  /**
+   * Returns the full version file path with given version number.
+   */
+  public static Path getVersionFilePath(HoodieTableMetaClient metaClient) {
+    return new Path(metaClient.getArchivePath(), VERSION_FILE_NAME);
+  }
+
+  /**
+   * List all the parquet manifest files.
+   */
+  public static FileStatus[] listAllManifestFiles(HoodieTableMetaClient metaClient) throws IOException {
+    return metaClient.getFs().listStatus(new Path(metaClient.getArchivePath()), getManifestFilePathFilter());
+  }
+
+  /**
+   * List all the parquet metadata files.
+   */
+  public static FileStatus[] listAllMetaFiles(HoodieTableMetaClient metaClient) throws IOException {
+    return metaClient.getFs().globStatus(
+        new Path(metaClient.getArchivePath() + "/*.parquet"));
+  }
+
+  /**
+   * Parse the snapshot version from the manifest file name.
+   */
+  public static int getManifestVersion(String fileName) {
+    return Integer.parseInt(fileName.split("_")[1]);
+  }
+
+  /**
+   * Parse the layer number from the file name.
+   */
+  public static int getFileLayer(String fileName) {
+    try {
+      Matcher fileMatcher = ARCHIVE_FILE_PATTERN.matcher(fileName);
+      if (fileMatcher.matches()) {
+        return Integer.parseInt(fileMatcher.group(3));
+      }
+    } catch (NumberFormatException e) {
+      // log and ignore any format warnings
+      LOG.warn("error getting file layout for archived file: " + fileName);
+    }
+
+    // return default value in case of any errors
+    return 0;
+  }
+
+  /**
+   * Parse the minimum instant time from the file name.
+   */
+  public static String getMinInstantTime(String fileName) {

Review Comment:
   instead of individual parsing methods, can introduce a POJO here, with getters. i.e 
   
   LSMFile class with min, max, level as fields?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1305300170


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/LSMTimeline.java:
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.common.model.HoodieLSMTimelineManifest;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * Represents the LSM Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *     \              /
+ *        \        /
+ *            |
+ *            V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *                                      |
+ *                                      V
+ *                                  _version_
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a compacted file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>
+ * Keeps only 3 valid snapshot versions for the reader, that means, a file is kept for at lest 3 archival trigger interval, for default configuration, it is 30 instants time span,
+ * which is far longer that the archived timeline loading time.
+ *
+ * <p><h3>Instants TTL</h3></p>
+ * The timeline reader only reads instants of last limited days. We will by default skip the instants from archived timeline that are generated long time ago.
+ */
+public class LSMTimeline {
+  private static final Logger LOG = LoggerFactory.getLogger(LSMTimeline.class);
+
+  private static final String VERSION_FILE_NAME = "_version_";    // _version_
+  private static final String MANIFEST_FILE_PREFIX = "manifest_"; // manifest_[N]
+
+  private static final String TEMP_FILE_SUFFIX = ".tmp";
+
+  private static final Pattern ARCHIVE_FILE_PATTERN =
+      Pattern.compile("^(\\d+)_(\\d+)_(\\d)\\.parquet");
+
+  // -------------------------------------------------------------------------
+  //  Utilities
+  // -------------------------------------------------------------------------
+  public static Schema getReadSchema(HoodieArchivedTimeline.LoadMode loadMode) {
+    switch (loadMode) {
+      case SLIM:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_SLIM_READ_SCHEMA;
+      case METADATA:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_READ_SCHEMA_WITH_METADATA;
+      case PLAN:
+        return ArchivedInstantReadSchemas.TIMELINE_LSM_READ_SCHEMA_WITH_PLAN;
+      default:
+        throw new AssertionError("Unexpected");
+    }
+  }
+
+  /**
+   * Returns whether the given file is located in the filter.
+   */
+  public static boolean isFileInRange(HoodieArchivedTimeline.TimeRangeFilter filter, String fileName) {
+    String minInstant = getMinInstantTime(fileName);
+    String maxInstant = getMaxInstantTime(fileName);
+    return filter.isInRange(minInstant) || filter.isInRange(maxInstant);
+  }
+
+  /**
+   * Returns the latest snapshot version.
+   */
+  public static int latestSnapshotVersion(HoodieTableMetaClient metaClient) throws IOException {
+    Path versionFilePath = getVersionFilePath(metaClient);
+    if (metaClient.getFs().exists(versionFilePath)) {
+      try {
+        Option<byte[]> content = FileIOUtils.readDataFromPath(metaClient.getFs(), versionFilePath);
+        if (content.isPresent()) {
+          return Integer.parseInt(new String(content.get(), StandardCharsets.UTF_8));
+        }
+      } catch (Exception e) {
+        // fallback to manifest file listing.
+        LOG.warn("Error reading version file {}", versionFilePath, e);
+      }
+    }
+    return allSnapshotVersions(metaClient).stream().max(Integer::compareTo).orElse(-1);
+  }
+
+  /**
+   * Returns all the valid snapshot versions.
+   */
+  public static List<Integer> allSnapshotVersions(HoodieTableMetaClient metaClient) throws IOException {
+    return Arrays.stream(metaClient.getFs().listStatus(new Path(metaClient.getArchivePath()), getManifestFilePathFilter()))
+        .map(fileStatus -> fileStatus.getPath().getName())
+        .map(LSMTimeline::getManifestVersion)
+        .collect(Collectors.toList());
+  }
+
+  /**
+   * Returns the latest snapshot metadata files.
+   */
+  public static HoodieLSMTimelineManifest latestSnapshotManifest(HoodieTableMetaClient metaClient) throws IOException {
+    int latestVersion = latestSnapshotVersion(metaClient);
+    return latestSnapshotManifest(metaClient, latestVersion);
+  }
+
+  /**
+   * Reads the file list from the manifest file for the latest snapshot.
+   */
+  public static HoodieLSMTimelineManifest latestSnapshotManifest(HoodieTableMetaClient metaClient, int latestVersion) {
+    if (latestVersion < 0) {
+      // there is no valid snapshot of the timeline.
+      return HoodieLSMTimelineManifest.EMPTY;
+    }
+    // read and deserialize the valid files.
+    byte[] content = FileIOUtils.readDataFromPath(metaClient.getFs(), getManifestFilePath(metaClient, latestVersion)).get();
+    try {
+      return HoodieLSMTimelineManifest.fromJsonString(new String(content, StandardCharsets.UTF_8), HoodieLSMTimelineManifest.class);
+    } catch (Exception e) {
+      throw new HoodieException("Error deserializing manifest entries", e);
+    }
+  }
+
+  /**
+   * Returns the full manifest file path with given version number.
+   */
+  public static Path getManifestFilePath(HoodieTableMetaClient metaClient, int snapshotVersion) {
+    return new Path(metaClient.getArchivePath(), MANIFEST_FILE_PREFIX + snapshotVersion);
+  }
+
+  /**
+   * Returns the full version file path with given version number.
+   */
+  public static Path getVersionFilePath(HoodieTableMetaClient metaClient) {
+    return new Path(metaClient.getArchivePath(), VERSION_FILE_NAME);
+  }
+
+  /**
+   * List all the parquet manifest files.
+   */
+  public static FileStatus[] listAllManifestFiles(HoodieTableMetaClient metaClient) throws IOException {
+    return metaClient.getFs().listStatus(new Path(metaClient.getArchivePath()), getManifestFilePathFilter());
+  }
+
+  /**
+   * List all the parquet metadata files.
+   */
+  public static FileStatus[] listAllMetaFiles(HoodieTableMetaClient metaClient) throws IOException {
+    return metaClient.getFs().globStatus(
+        new Path(metaClient.getArchivePath() + "/*.parquet"));
+  }
+
+  /**
+   * Parse the snapshot version from the manifest file name.
+   */
+  public static int getManifestVersion(String fileName) {
+    return Integer.parseInt(fileName.split("_")[1]);
+  }
+
+  /**
+   * Parse the layer number from the file name.
+   */
+  public static int getFileLayer(String fileName) {
+    try {
+      Matcher fileMatcher = ARCHIVE_FILE_PATTERN.matcher(fileName);
+      if (fileMatcher.matches()) {
+        return Integer.parseInt(fileMatcher.group(3));
+      }
+    } catch (NumberFormatException e) {
+      // log and ignore any format warnings
+      LOG.warn("error getting file layout for archived file: " + fileName);
+    }
+
+    // return default value in case of any errors
+    return 0;
+  }
+
+  /**
+   * Parse the minimum instant time from the file name.
+   */
+  public static String getMinInstantTime(String fileName) {
+    Matcher fileMatcher = ARCHIVE_FILE_PATTERN.matcher(fileName);
+    if (fileMatcher.matches()) {
+      return fileMatcher.group(1);
+    } else {
+      throw new HoodieException("Unexpected archival file name: " + fileName);
+    }
+  }
+
+  /**
+   * Parse the maximum instant time from the file name.
+   */
+  public static String getMaxInstantTime(String fileName) {

Review Comment:
   Tests added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1693079449

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461",
       "triggerID" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19463",
       "triggerID" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 2288afba514c322841e5479a7fc7f6cdc8bf8b2d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461) 
   * 613a47bc53981cd9bc4c1fe87c30713fa0f5f2be Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19463) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1657991575

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * a7f8558aaffdaab4850780224e1385c3e682372a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824) 
   * 9e7266dffff31156421a129d11dbde82be2fc4fd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280271723


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>
+ * Keeps only 3 valid snapshot versions for the reader, that means, a file is kept for at lest 3 archival trigger interval, for default configuration, it is 30 instants time span,
+ * which is far longer that the archived timeline loading time.
+ *
+ * <p><h3>Instants TTL</h3></p>

Review Comment:
   +1 too, by default we can eagerly load 3 ~ 7 days of instants into the memory for fast look up of completion time, if there is any instant out of this range, do a lazy loading for it, the performance should be not too bad because we have a data skipping on files.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1661869288

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 6ab7d4bd2696988e3784291879d1f713d6a9c98d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978) 
   * 4ade37c10c908c0422915aaa489208e6ee62bb0d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1659477292

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 9e7266dffff31156421a129d11dbde82be2fc4fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919) 
   * 1874d4fadea7db5be55f0fd938276eb594258357 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1694129217

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461",
       "triggerID" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19463",
       "triggerID" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19467",
       "triggerID" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d688280b7fde224d75843893290db559bcc169f5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d688280b7fde224d75843893290db559bcc169f5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 1739fa36bb1e777abb6f85d429a2391eacec47ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19467) 
   * d688280b7fde224d75843893290db559bcc169f5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1693095319

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461",
       "triggerID" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19463",
       "triggerID" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 2288afba514c322841e5479a7fc7f6cdc8bf8b2d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461) 
   * 613a47bc53981cd9bc4c1fe87c30713fa0f5f2be Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19463) 
   * 1739fa36bb1e777abb6f85d429a2391eacec47ce UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1694177886

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461",
       "triggerID" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19463",
       "triggerID" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19467",
       "triggerID" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d688280b7fde224d75843893290db559bcc169f5",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19480",
       "triggerID" : "d688280b7fde224d75843893290db559bcc169f5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * d688280b7fde224d75843893290db559bcc169f5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19480) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6539] New LSM tree style archived timeline [hudi]

Posted by "BruceKellan (via GitHub)" <gi...@apache.org>.
BruceKellan commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1392239045


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -164,230 +101,32 @@ public boolean archiveIfRequired(HoodieEngineContext context, boolean acquireLoc
         // there is no owner or instant time per se for archival.
         txnManager.beginTransaction(Option.empty(), Option.empty());
       }
-      List<HoodieInstant> instantsToArchive = getInstantsToArchive().collect(Collectors.toList());
-      verifyLastMergeArchiveFilesIfNecessary(context);
+      // Sort again because the cleaning and rollback instants could break the sequence.
+      List<ActiveAction> instantsToArchive = getInstantsToArchive().sorted().collect(Collectors.toList());
       boolean success = true;
       if (!instantsToArchive.isEmpty()) {
-        this.writer = openWriter();
         LOG.info("Archiving instants " + instantsToArchive);
-        archive(context, instantsToArchive);
+        Consumer<Exception> exceptionHandler = e -> {
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw new HoodieException(e);
+          }
+        };
+        this.timelineWriter.write(instantsToArchive, Option.of(action -> deleteAnyLeftOverMarkers(context, action)), Option.of(exceptionHandler));
         LOG.info("Deleting archived instants " + instantsToArchive);
         success = deleteArchivedInstants(instantsToArchive, context);

Review Comment:
   if we need to consider how to handle deletion failure exception?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287892832


##########
hudi-common/src/main/java/org/apache/hudi/common/util/ArchivedInstantReadSchemas.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.avro.Schema;
+
+/**
+ * Avro schema for different archived instant read cases.
+ */
+public abstract class ArchivedInstantReadSchemas {

Review Comment:
   We already have a avro model file named `HoodieArchivedInstant`, no need to declare read schema for every use cases, a JAVA constant should be enough.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288154950


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();

Review Comment:
   Rename to: `hoodie.timeline.compaction.batch.size`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1671117746

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 57c1b843608a9b63d143ead5dd5168613bb13969 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027) 
   * 803df61d0d04f7e7403d1177325a365e9bbafab5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6539] New LSM tree style archived timeline [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1776308765

   > Hello, does the master branch now support lsm format merge? @danny0405
   
   No, only the archived timeline uses LSM layout for instants access.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287894094


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/FileCreateUtils.java:
##########
@@ -278,29 +278,16 @@ public static void createRestoreFile(String basePath, String instantTime, Hoodie
     createMetaFile(basePath, instantTime, HoodieTimeline.RESTORE_ACTION, serializeRestoreMetadata(hoodieRestoreMetadata).get());
   }
 
-  private static void createAuxiliaryMetaFile(String basePath, String instantTime, String suffix) throws IOException {

Review Comment:
   Yes, this is only used in tests and actually a misuse here, so I just remove it and fix the tests. (the original test wants to add commit meta file but it mistakenly adds in a wrong place, and the test is vary hacky to make pass.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287870522


##########
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java:
##########
@@ -846,6 +846,50 @@ public static List<FileStatus> getFileStatusAtLevel(
     return result;
   }
 
+  /**
+   * Renames the source file to destination, using the provided file system. If the rename failed,
+   * an attempt will be made to delete the source file.
+   */
+  public static void renameFile(FileSystem fs, Path src, Path dst) {
+    try {
+      if (fs.exists(dst)) {
+        throw new HoodieIOException("Target file " + dst + " already exists");
+      }
+      if (!fs.rename(src, dst)) {
+        handleException(fs, src, dst);
+      }
+    } catch (IOException e) {
+      handleException(fs, src, dst);
+    }
+  }
+
+  /**
+   * Handles the exception for renaming.
+   */
+  private static void handleException(FileSystem fs, Path src, Path dst) {
+    HoodieIOException ioe = new HoodieIOException("Failed to rename " + src + " to " + dst);
+    IOException ex = tryDelete(fs, src);
+    if (ex != null) {
+      ioe.addSuppressed(ex);
+    }
+    throw ioe;
+  }
+
+  /**
+   * Deletes the file from the file system.
+   *
+   * @param path the file to be deleted.
+   * @return IOException caught, if any. Null otherwise.
+   */
+  private static IOException tryDelete(FileSystem fs, Path path) {

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287874128


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java:
##########
@@ -493,17 +224,18 @@ private Stream<HoodieInstant> getCommitInstantsToArchive() throws IOException {
     }
   }
 
-  private Stream<HoodieInstant> getInstantsToArchive() throws IOException {
-    Stream<HoodieInstant> instants = Stream.concat(getCleanInstantsToArchive(), getCommitInstantsToArchive());
+  private Stream<ActiveInstant> getInstantsToArchive() throws IOException {

Review Comment:
   Yeah, confirmed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1694131258

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461",
       "triggerID" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19463",
       "triggerID" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19467",
       "triggerID" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d688280b7fde224d75843893290db559bcc169f5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19480",
       "triggerID" : "d688280b7fde224d75843893290db559bcc169f5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 1739fa36bb1e777abb6f85d429a2391eacec47ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19467) 
   * d688280b7fde224d75843893290db559bcc169f5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19480) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1649344787

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * c281ded6d554350dfe362cce496d6d72cfe0bbbe Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822) 
   * a7f8558aaffdaab4850780224e1385c3e682372a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1648265426

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62001bbc6c5d9306f95d4bca3dd2bada3ca5c898 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1657919730

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * a7f8558aaffdaab4850780224e1385c3e682372a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824) 
   * 9e7266dffff31156421a129d11dbde82be2fc4fd UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280075324


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>

Review Comment:
   I was thinking why not just read the manifest with the highest timestamp or version? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1278788392


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveInstant.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * A combination of instants covering action states: requested, inflight, completed.
+ */
+public class ActiveInstant implements Serializable, Comparable<ActiveInstant> {
+  private final HoodieInstant requested;
+  private final HoodieInstant inflight;
+  private final HoodieInstant completed;
+
+  /**
+   * The constructor.
+   */
+  protected ActiveInstant(@Nullable HoodieInstant requested, @Nullable HoodieInstant inflight, HoodieInstant completed) {
+    this.requested = requested;
+    this.inflight = inflight;
+    this.completed = completed;
+  }
+
+  public static ActiveInstant fromInstants(List<HoodieInstant> instants) {
+    ValidationUtils.checkArgument(instants.size() <= 3);
+    HoodieInstant requested = null;
+    HoodieInstant inflight = null;
+    HoodieInstant completed = null;
+    for (HoodieInstant instant : instants) {
+      if (instant.isRequested()) {
+        requested = instant;
+      } else if (instant.isInflight()) {
+        inflight = instant;
+      } else {
+        completed = instant;
+      }
+    }
+    return new ActiveInstant(requested, inflight, Objects.requireNonNull(completed));
+  }
+
+  public List<HoodieInstant> getPendingInstants() {
+    List<HoodieInstant> instants = new ArrayList<>(2);
+    if (this.requested != null) {
+      instants.add(this.requested);
+    }
+    if (this.inflight != null) {
+      instants.add(this.inflight);
+    }
+    return instants;
+  }
+
+  public HoodieInstant getCompleted() {
+    return completed;
+  }
+
+  public String getAction() {
+    return this.completed.getAction();
+  }
+
+  /**
+   * A COMPACTION action eventually becomes COMMIT when completed.

Review Comment:
   Personally I'm +1 for keeping the action of `compaction` and `log_compaction` just as it is, this avoids many ambiguilties, but I kind of think it should be a separate topic, we need a discuss whether to reuse the action for all kinds of table services: compaction, log_compaction, clustering, etc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1278789233


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveInstant.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * A combination of instants covering action states: requested, inflight, completed.
+ */
+public class ActiveInstant implements Serializable, Comparable<ActiveInstant> {

Review Comment:
   It's just a triple of an instant with 3 different states, maybe we can come out with a better name for it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1662857335

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 4ade37c10c908c0422915aaa489208e6ee62bb0d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1693106634

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461",
       "triggerID" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19463",
       "triggerID" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19467",
       "triggerID" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 613a47bc53981cd9bc4c1fe87c30713fa0f5f2be Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19463) 
   * 1739fa36bb1e777abb6f85d429a2391eacec47ce Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19467) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1660774973

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 9a2afc24ea8552ec6869f9d5819a00c78067a4e6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1278797024


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>

Review Comment:
   The classic way is there is one version pointer file, which always records the latest version id of the LSM tree, the writer needs to atomically updates this version pointer file each time for any modifications to the LSM tree.
   
   The reader reads the version pointer file first, if there is any exception happens during the reading process, fallback to listing the manifest files to fetch the latest version.
   
   The reason I didn't do that is this way can not really keep the atomicity, the reader has very low opportunity to read the corrupt version file/manifest file actually, but in general, we can migrate to that because it can avoids the file listing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1278790941


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>

Review Comment:
   Corrent, to be more accurate, it keeps the list of all the files in the latest snapshot. The LSM tree itself has multiple versions.



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>

Review Comment:
   Correct, to be more accurate, it keeps the list of all the files in the latest snapshot. The LSM tree itself has multiple versions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1278793361


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.

Review Comment:
   Now for multi-writer, we alredy have a explicit lock guard for the archiver. The lsm merge now is an inline action right after the write, we might need to support async compaction in the future ? I have no idea yet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1658376108

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 9e7266dffff31156421a129d11dbde82be2fc4fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280276081


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>

Review Comment:
   Either is okay, the version number based cleaning works better when the timeline is committed more frequently, because we do not need timetravel queries on the timeline, the cleaning can be more redical.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280082319


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.

Review Comment:
   yeah for now we assume this is within a lock and done inline. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1663317607

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 4ade37c10c908c0422915aaa489208e6ee62bb0d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997) 
   * 57c1b843608a9b63d143ead5dd5168613bb13969 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287850688


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java:
##########
@@ -164,230 +100,27 @@ public boolean archiveIfRequired(HoodieEngineContext context, boolean acquireLoc
         // there is no owner or instant time per se for archival.
         txnManager.beginTransaction(Option.empty(), Option.empty());
       }
-      List<HoodieInstant> instantsToArchive = getInstantsToArchive().collect(Collectors.toList());
-      verifyLastMergeArchiveFilesIfNecessary(context);
+      // Sort again because the cleaning and rollback instants could break the sequence.
+      List<ActiveInstant> instantsToArchive = getInstantsToArchive().sorted().collect(Collectors.toList());
       boolean success = true;
       if (!instantsToArchive.isEmpty()) {
-        this.writer = openWriter();
         LOG.info("Archiving instants " + instantsToArchive);
-        archive(context, instantsToArchive);
+        this.timelineWriter.write(context, instantsToArchive);
         LOG.info("Deleting archived instants " + instantsToArchive);
         success = deleteArchivedInstants(instantsToArchive, context);
+        // triggers compaction and cleaning only after archiving action
+        this.timelineWriter.compactAndClean(context);
       } else {
         LOG.info("No Instants to archive");
       }
-
-      if (shouldMergeSmallArchiveFiles()) {

Review Comment:
   Kind of, it is replaced with the new compaction, but we still keep one config option, that is the number of bacth files for each compaction input source.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288092558


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -184,203 +222,327 @@ public HoodieArchivedTimeline reload() {
     return new HoodieArchivedTimeline(metaClient);
   }
 
-  private HoodieInstant readCommit(GenericRecord record, boolean loadDetails) {
-    final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
-    final String action = record.get(ACTION_TYPE_KEY).toString();
-    final String stateTransitionTime = (String) record.get(STATE_TRANSITION_TIME);
-    if (loadDetails) {
-      getMetadataKey(action).map(key -> {
-        Object actionData = record.get(key);
-        if (actionData != null) {
-          if (action.equals(HoodieTimeline.COMPACTION_ACTION)) {
-            this.readCommits.put(instantTime, HoodieAvroUtils.indexedRecordToBytes((IndexedRecord) actionData));
-          } else {
-            this.readCommits.put(instantTime, actionData.toString().getBytes(StandardCharsets.UTF_8));
-          }
+  private HoodieInstant readCommit(GenericRecord record, LoadMode loadMode) {
+    final String instantTime = record.get(INSTANT_TIME_ARCHIVED_META_FIELD).toString();
+    final String action = record.get(ACTION_ARCHIVED_META_FIELD).toString();
+    final String completionTime = record.get(COMPLETION_TIME_ARCHIVED_META_FIELD).toString();
+    loadInstantDetails(record, instantTime, loadMode);
+    return new HoodieInstant(HoodieInstant.State.COMPLETED, action, instantTime, completionTime);
+  }
+
+  private void loadInstantDetails(GenericRecord record, String instantTime, LoadMode loadMode) {
+    switch (loadMode) {
+      case COMMIT_META:
+        ByteBuffer commitMeta = (ByteBuffer) record.get(METADATA_ARCHIVED_META_FIELD);
+        if (commitMeta != null) {
+          // in case the entry comes from an empty completed meta file
+          this.readCommits.put(instantTime, commitMeta.array());
         }
-        return null;
-      });
-    }
-    return new HoodieInstant(HoodieInstant.State.valueOf(record.get(ACTION_STATE).toString()), action,
-        instantTime, stateTransitionTime);
-  }
-
-  @Nonnull
-  private Option<String> getMetadataKey(String action) {
-    switch (action) {
-      case HoodieTimeline.CLEAN_ACTION:
-        return Option.of("hoodieCleanMetadata");
-      case HoodieTimeline.COMMIT_ACTION:
-      case HoodieTimeline.DELTA_COMMIT_ACTION:
-        return Option.of("hoodieCommitMetadata");
-      case HoodieTimeline.ROLLBACK_ACTION:
-        return Option.of("hoodieRollbackMetadata");
-      case HoodieTimeline.SAVEPOINT_ACTION:
-        return Option.of("hoodieSavePointMetadata");
-      case HoodieTimeline.COMPACTION_ACTION:
-      case HoodieTimeline.LOG_COMPACTION_ACTION:
-        return Option.of("hoodieCompactionPlan");
-      case HoodieTimeline.REPLACE_COMMIT_ACTION:
-        return Option.of("hoodieReplaceCommitMetadata");
-      case HoodieTimeline.INDEXING_ACTION:
-        return Option.of("hoodieIndexCommitMetadata");
+        break;
+      case PLAN:
+        ByteBuffer plan = (ByteBuffer) record.get(PLAN_ARCHIVED_META_FIELD);
+        if (plan != null) {

Review Comment:
   Yeah, the plan is only useful for CLI presentation now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1671129502

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 57c1b843608a9b63d143ead5dd5168613bb13969 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027) 
   * 803df61d0d04f7e7403d1177325a365e9bbafab5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288171581


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {
+      // 2. sort files by min instant time (implies ascending chronological order)
+      files.sort(HoodieArchivedManifest.FileEntry::compareTo);
+      List<String> candidateFiles = getCandidateFiles(files, compactionBatchSize);
+      if (candidateFiles.size() < 2) {
+        // the file is too large to compact, returns early.
+        return Option.empty();
+      }
+      String compactedFileName = compactedFileName(candidateFiles);
+
+      // 3. compaction
+      compactArchiveFiles(candidateFiles, compactedFileName);
+      // 4. update the manifest file
+      updateManifest(candidateFiles, compactedFileName);
+      LOG.info("Finishes compaction of archive files: " + candidateFiles);
+      return Option.of(compactedFileName);
+    }
+    return Option.empty();
+  }
+
+  public void compactArchiveFiles(List<String> candidateFiles, String compactedFileName) {
+    LOG.info("Starting to merge small archive files.");
+    try (HoodieFileWriter writer = openWriter(new Path(metaClient.getArchivePath(), compactedFileName))) {
+      for (String fileName : candidateFiles) {
+        // Read the archived file
+        try (HoodieAvroParquetReader reader = (HoodieAvroParquetReader) HoodieFileReaderFactory.getReaderFactory(HoodieRecord.HoodieRecordType.AVRO)
+            .getFileReader(metaClient.getHadoopConf(), new Path(metaClient.getArchivePath(), fileName))) {
+          // Read the meta entry
+          try (ClosableIterator<IndexedRecord> iterator = reader.getIndexedRecordIterator(HoodieArchivedInstant.getClassSchema(), HoodieArchivedInstant.getClassSchema())) {
+            while (iterator.hasNext()) {
+              IndexedRecord record = iterator.next();
+              writer.write(record.get(0).toString(), new HoodieAvroIndexedRecord(record), HoodieArchivedInstant.getClassSchema());
+            }
+          }
+        }
+      }
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to merge small archive files", e);
+    }
+    LOG.info("Success to merge small archive files.");
+  }
+
+  /**
+   * Checks whether there is any unfinished compaction operation.
+   *
+   * @param context HoodieEngineContext used for parallelize to delete small archive files if necessary.
+   */
+  public void clean(HoodieEngineContext context, int compactedVersions) throws IOException {
+    // if there are more than 3 version of snapshots, clean the oldest files.
+    List<Integer> allSnapshotVersions = HoodieArchivedTimeline.allSnapshotVersions(metaClient);
+    int numVersionsToKeep = 3 + compactedVersions; // should make the threshold configurable.
+    if (allSnapshotVersions.size() > numVersionsToKeep) {
+      allSnapshotVersions.sort((v1, v2) -> v2 - v1);
+      List<Integer> versionsToKeep = allSnapshotVersions.subList(0, numVersionsToKeep);
+      Set<String> filesToKeep = versionsToKeep.stream()
+          .flatMap(version -> HoodieArchivedTimeline.latestSnapshotManifest(metaClient, version).getFileNames().stream())
+          .collect(Collectors.toSet());
+      // delete the manifest file first
+      List<String> manifestFilesToClean = new ArrayList<>();
+      Arrays.stream(HoodieArchivedTimeline.listAllManifestFiles(metaClient)).forEach(fileStatus -> {
+        if (!versionsToKeep.contains(HoodieArchivedTimeline.getManifestVersion(fileStatus.getPath().getName()))) {
+          manifestFilesToClean.add(fileStatus.getPath().toString());
+        }
+      });
+      deleteFilesParallelize(metaClient, manifestFilesToClean, context, false);

Review Comment:
   then the next clean can still take over.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288113616


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;

Review Comment:
   I guess it deserves a separate task.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288140061


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {

Review Comment:
   We can add when necessary, it is quite easy to extend.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288155900


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {

Review Comment:
   First entries based but also limited by the size.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6539] New LSM tree style archived timeline [hudi]

Posted by "BruceKellan (via GitHub)" <gi...@apache.org>.
BruceKellan commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1393559840


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -164,230 +101,32 @@ public boolean archiveIfRequired(HoodieEngineContext context, boolean acquireLoc
         // there is no owner or instant time per se for archival.
         txnManager.beginTransaction(Option.empty(), Option.empty());
       }
-      List<HoodieInstant> instantsToArchive = getInstantsToArchive().collect(Collectors.toList());
-      verifyLastMergeArchiveFilesIfNecessary(context);
+      // Sort again because the cleaning and rollback instants could break the sequence.
+      List<ActiveAction> instantsToArchive = getInstantsToArchive().sorted().collect(Collectors.toList());
       boolean success = true;
       if (!instantsToArchive.isEmpty()) {
-        this.writer = openWriter();
         LOG.info("Archiving instants " + instantsToArchive);
-        archive(context, instantsToArchive);
+        Consumer<Exception> exceptionHandler = e -> {
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw new HoodieException(e);
+          }
+        };
+        this.timelineWriter.write(instantsToArchive, Option.of(action -> deleteAnyLeftOverMarkers(context, action)), Option.of(exceptionHandler));
         LOG.info("Deleting archived instants " + instantsToArchive);
         success = deleteArchivedInstants(instantsToArchive, context);

Review Comment:
   No more doubts.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288178605


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;

Review Comment:
   We have no limit for layers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287854438


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveInstant.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * A combination of instants covering action states: requested, inflight, completed.
+ */
+public class ActiveInstant implements Serializable, Comparable<ActiveInstant> {

Review Comment:
   +1 for `ActiveAction`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287865885


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>

Review Comment:
   Update: 
   
   1. we now only have 1 version hint file, whose content is the latest version number.
   2. for HDFS, we will do a rename from a temp file to the final file, for both the manifest and version file.
   3. for s3, do a direct write because object storage itself has atomicity for write operation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288167338


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ArchivedTimelineWriter.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.io.storage.HoodieFileWriter;
+import org.apache.hudi.io.storage.HoodieFileWriterFactory;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.marker.WriteMarkers;
+import org.apache.hudi.table.marker.WriteMarkersFactory;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * An archived timeline writer which organizes the files as an LSM tree.
+ */
+public class ArchivedTimelineWriter {
+  private static final Logger LOG = LoggerFactory.getLogger(ArchivedTimelineWriter.class);
+
+  private final HoodieWriteConfig config;
+  private final HoodieTable<?, ?, ?, ?> table;
+  private final HoodieTableMetaClient metaClient;
+
+  private HoodieWriteConfig writeConfig;
+
+  private ArchivedTimelineWriter(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    this.config = config;
+    this.table = table;
+    this.metaClient = table.getMetaClient();
+  }
+
+  public static ArchivedTimelineWriter getInstance(HoodieWriteConfig config, HoodieTable<?, ?, ?, ?> table) {
+    return new ArchivedTimelineWriter(config, table);
+  }
+
+  public void write(HoodieEngineContext context, List<ActiveInstant> instants) throws HoodieCommitException {
+    Path filePath = new Path(metaClient.getArchivePath(),
+        newFileName(instants.get(0).getInstantTime(), instants.get(instants.size() - 1).getInstantTime(), HoodieArchivedTimeline.FILE_LAYER_ZERO));
+    try (HoodieFileWriter writer = openWriter(filePath)) {
+      Schema wrapperSchema = HoodieArchivedInstant.getClassSchema();
+      LOG.info("Archiving schema " + wrapperSchema.toString());
+      for (ActiveInstant triple : instants) {
+        try {
+          deleteAnyLeftOverMarkers(context, triple);
+          // in local FS and HDFS, there could be empty completed instants due to crash.
+          final HoodieArchivedInstant metaEntry = MetadataConversionUtils.createArchivedInstant(triple, metaClient);
+          writer.write(metaEntry.getInstantTime(), new HoodieAvroIndexedRecord(metaEntry), wrapperSchema);
+        } catch (Exception e) {
+          LOG.error("Failed to archive instant: " + triple.getInstantTime(), e);
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw e;
+          }
+        }
+      }
+      updateManifest(filePath.getName());
+    } catch (Exception e) {
+      throw new HoodieCommitException("Failed to archive commits", e);
+    }
+  }
+
+  public void updateManifest(String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy();
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  public void updateManifest(List<String> filesToRemove, String fileToAdd) throws IOException {
+    // 1. read the latest manifest version file;
+    // 2. read the latest manifest file for valid files;
+    // 3. remove files to the existing file list from step2;
+    // 4. add this new file to the existing file list from step2.
+    int latestVersion = HoodieArchivedTimeline.latestSnapshotVersion(metaClient);
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient, latestVersion);
+    HoodieArchivedManifest newManifest = latestManifest.copy(filesToRemove);
+    newManifest.addFile(getFileEntry(fileToAdd));
+    createManifestFile(newManifest, latestVersion);
+  }
+
+  private void createManifestFile(HoodieArchivedManifest manifest, int currentVersion) throws IOException {
+    byte[] content = manifest.toJsonString().getBytes(StandardCharsets.UTF_8);
+    // version starts from 1 and increases monotonically
+    int newVersion = currentVersion < 0 ? 1 : currentVersion + 1;
+    // create manifest file
+    final Path manifestFilePath = HoodieArchivedTimeline.getManifestFilePath(metaClient, newVersion);
+    metaClient.getFs().createImmutableFileInPath(manifestFilePath, Option.of(content));
+    // update version file
+    updateVersionFile(newVersion);
+  }
+
+  private void updateVersionFile(int newVersion) throws IOException {
+    byte[] content = (String.valueOf(newVersion)).getBytes(StandardCharsets.UTF_8);
+    final Path versionFilePath = HoodieArchivedTimeline.getVersionFilePath(metaClient);
+    metaClient.getFs().delete(versionFilePath, false);
+    metaClient.getFs().createImmutableFileInPath(versionFilePath, Option.of(content));
+  }
+
+  /**
+   * Compacts the small archive files.
+   *
+   * <p>The parquet naming convention is:
+   *
+   * <pre>${min_instant}_${max_instant}_${level}.parquet</pre>
+   *
+   * <p>The 'min_instant' and 'max_instant' represent the instant time range of the parquet file.
+   * The 'level' represents the number of the level where the file is located, currently we
+   * have no limit for the number of layers.
+   *
+   * <p>These archive parquet files composite as an LSM tree layout, one parquet file contains
+   * a consecutive timestamp instant metadata entries. Different parquet files may have
+   * overlapping with the instant time ranges.
+   *
+   * <pre>
+   *   t1_t2_0.parquet, t3_t4_0.parquet, ... t5_t6_0.parquet       L0 layer
+   *                          \            /
+   *                             \     /
+   *                                |
+   *                                V
+   *                          t3_t6_1.parquet                      L1 layer
+   * </pre>
+   *
+   * <p>Compaction and cleaning: once the files number exceed a threshold(now constant 10) N,
+   * the oldest N files are then replaced with a compacted file in the next layer.
+   * A cleaning action is triggered right after the compaction.
+   *
+   * @param context HoodieEngineContext
+   */
+  @VisibleForTesting
+  public void compactAndClean(HoodieEngineContext context) throws IOException {
+    // 1. List all the latest snapshot files
+    HoodieArchivedManifest latestManifest = HoodieArchivedTimeline.latestSnapshotManifest(metaClient);
+    int layer = 0;
+    // 2. triggers the compaction for L0
+    Option<String> compactedFileName = doCompact(latestManifest, layer);
+    while (compactedFileName.isPresent()) {
+      // 3. once a compaction had been executed for the current layer,
+      // continues to trigger compaction for the next layer.
+      latestManifest.addFile(getFileEntry(compactedFileName.get()));
+      compactedFileName = doCompact(latestManifest, ++layer);
+    }
+
+    // cleaning
+    clean(context, layer);
+  }
+
+  private Option<String> doCompact(HoodieArchivedManifest manifest, int layer) throws IOException {
+    // 1. list all the files that belong to current layer
+    List<HoodieArchivedManifest.FileEntry> files = manifest.getFiles()
+        .stream().filter(file -> HoodieArchivedTimeline.isFileFromLayer(file.getFileName(), layer)).collect(Collectors.toList());
+
+    int compactionBatchSize = config.getArchiveMergeFilesBatchSize();
+
+    if (files.size() >= compactionBatchSize) {
+      // 2. sort files by min instant time (implies ascending chronological order)
+      files.sort(HoodieArchivedManifest.FileEntry::compareTo);
+      List<String> candidateFiles = getCandidateFiles(files, compactionBatchSize);
+      if (candidateFiles.size() < 2) {
+        // the file is too large to compact, returns early.
+        return Option.empty();
+      }
+      String compactedFileName = compactedFileName(candidateFiles);
+
+      // 3. compaction
+      compactArchiveFiles(candidateFiles, compactedFileName);
+      // 4. update the manifest file
+      updateManifest(candidateFiles, compactedFileName);
+      LOG.info("Finishes compaction of archive files: " + candidateFiles);
+      return Option.of(compactedFileName);
+    }
+    return Option.empty();
+  }
+
+  public void compactArchiveFiles(List<String> candidateFiles, String compactedFileName) {
+    LOG.info("Starting to merge small archive files.");
+    try (HoodieFileWriter writer = openWriter(new Path(metaClient.getArchivePath(), compactedFileName))) {

Review Comment:
   Not sure it deserves to do that, the sort merge costs much memory.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287874876


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java:
##########
@@ -603,53 +327,4 @@ private boolean deleteArchivedInstants(List<HoodieInstant> archivedInstants, Hoo
 
     return true;
   }
-
-  public void archive(HoodieEngineContext context, List<HoodieInstant> instants) throws HoodieCommitException {
-    try {
-      Schema wrapperSchema = HoodieArchivedMetaEntry.getClassSchema();
-      LOG.info("Wrapper schema " + wrapperSchema.toString());
-      List<IndexedRecord> records = new ArrayList<>();
-      for (HoodieInstant hoodieInstant : instants) {
-        try {
-          deleteAnyLeftOverMarkers(context, hoodieInstant);
-          records.add(convertToAvroRecord(hoodieInstant));
-          if (records.size() >= this.config.getCommitArchivalBatchSize()) {
-            writeToFile(wrapperSchema, records);
-          }
-        } catch (Exception e) {
-          LOG.error("Failed to archive commits, .commit file: " + hoodieInstant.getFileName(), e);
-          if (this.config.isFailOnTimelineArchivingEnabled()) {
-            throw e;
-          }
-        }
-      }
-      writeToFile(wrapperSchema, records);
-    } catch (Exception e) {
-      throw new HoodieCommitException("Failed to archive commits", e);
-    }
-  }
-
-  private void deleteAnyLeftOverMarkers(HoodieEngineContext context, HoodieInstant instant) {

Review Comment:
   To `ArchivedTimelineWriter`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1671364778

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 803df61d0d04f7e7403d1177325a365e9bbafab5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1649319585

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9889e40cdf17f6f24ddefff010a063d4dd2c58e7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820) 
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * c281ded6d554350dfe362cce496d6d72cfe0bbbe Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822) 
   * a7f8558aaffdaab4850780224e1385c3e682372a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1649243715

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62001bbc6c5d9306f95d4bca3dd2bada3ca5c898 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804) 
   * 9889e40cdf17f6f24ddefff010a063d4dd2c58e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820) 
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1278789233


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveInstant.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+
+import javax.annotation.Nullable;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * A combination of instants covering action states: requested, inflight, completed.
+ */
+public class ActiveInstant implements Serializable, Comparable<ActiveInstant> {

Review Comment:
   It's just a triple of a instant with 3 different states, maybe we can come out with a better name for it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280085427


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>

Review Comment:
   lets include the info we need for e.g timestamp of when a version was created? to help retain x hours of versions. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280086564


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,

Review Comment:
   Should we do some schemaful file to store manifests? parquet/avro?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280074068


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,

Review Comment:
   size makes sense, do we need the size info in the manifest as well then/



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,

Review Comment:
   size makes sense, do we need the size info in the manifest as well then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280074900


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>

Review Comment:
   To understand this better, 
   
   for distributed file system like HDFS: this helps reader from reading any partially written MANIFEST file? 
   for cloud storage: PUTs are atomic, so reader will not see any partially written MANIFEST file?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vinothchandar commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280082204


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>

Review Comment:
   I agree with the version file and it points to latest file list/manifest



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1693493118

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461",
       "triggerID" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19463",
       "triggerID" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19467",
       "triggerID" : "1739fa36bb1e777abb6f85d429a2391eacec47ce",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 1739fa36bb1e777abb6f85d429a2391eacec47ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19467) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1278791765


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>

Review Comment:
   It behaves like a MARKER file for the manifest file, a version file handle indicates the version of snapshot is now complete for reader view, the reader list all the valid version files to fetch the valid versions of the current timeline.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1661681506

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 6ab7d4bd2696988e3784291879d1f713d6a9c98d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1659833152

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 9e7266dffff31156421a129d11dbde82be2fc4fd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919) 
   * 1874d4fadea7db5be55f0fd938276eb594258357 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945) 
   * 9a2afc24ea8552ec6869f9d5819a00c78067a4e6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1659916719

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 1874d4fadea7db5be55f0fd938276eb594258357 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945) 
   * 9a2afc24ea8552ec6869f9d5819a00c78067a4e6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1663323825

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 4ade37c10c908c0422915aaa489208e6ee62bb0d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997) 
   * 57c1b843608a9b63d143ead5dd5168613bb13969 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1288283343


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,127 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
+import org.apache.hudi.common.model.HoodieArchivedManifest;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
+import java.nio.ByteBuffer;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
-import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *     \              /
+ *        \        /
+ *            |
+ *            V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *                                      |
+ *                                      V
+ *                                  _version_
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a compacted file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1305108862


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTimelineArchiver.java:
##########
@@ -424,9 +159,7 @@ private Stream<HoodieInstant> getCommitInstantsToArchive() throws IOException {
                   LESSER_THAN, oldestPendingInstant.get().getTimestamp())).findFirst());
       // Check if the completed instant is higher than the oldest inflight instant
       // in that case update the oldestCommitToRetain to oldestInflight commit time.
-      if (!completedCommitBeforeOldestPendingInstant.isPresent()
-          || HoodieTimeline.compareTimestamps(oldestPendingInstant.get().getTimestamp(),
-          LESSER_THAN, completedCommitBeforeOldestPendingInstant.get().getTimestamp())) {
+      if (!completedCommitBeforeOldestPendingInstant.isPresent()) {

Review Comment:
   Because the check
   ```java
   HoodieTimeline.compareTimestamps(oldestPendingInstant.get().getTimestamp(),
             LESSER_THAN, completedCommitBeforeOldestPendingInstant.get().getTimestamp())
   ```
   
   is always false. I can revert the change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1305073510


##########
hudi-common/src/main/avro/HoodieLSMTimelineInstant.avsc:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+   "type":"record",
+   "name":"HoodieLSMTimelineInstant",
+   "namespace":"org.apache.hudi.avro.model",
+   "fields":[
+      {
+         "name":"instantTime",
+         "type":["null","string"],
+         "default": null
+      },
+      {
+         "name":"completionTime",
+         "type":["null","string"],
+         "default": null
+      },
+      {
+         "name":"action",
+         "type":["null","string"],
+         "default": null
+      },
+      {
+         "name":"metadata",
+         "type":["null", "bytes"],
+         "default": null
+      },
+      {
+         "name":"plan",
+         "type":["null", "bytes"],

Review Comment:
   Yeah, bytes is more effocient. Fired a follow up JIRA: https://issues.apache.org/jira/browse/HUDI-6747



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1693016032

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18822",
       "triggerID" : "c281ded6d554350dfe362cce496d6d72cfe0bbbe",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18824",
       "triggerID" : "a7f8558aaffdaab4850780224e1385c3e682372a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18919",
       "triggerID" : "9e7266dffff31156421a129d11dbde82be2fc4fd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18945",
       "triggerID" : "1874d4fadea7db5be55f0fd938276eb594258357",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18955",
       "triggerID" : "9a2afc24ea8552ec6869f9d5819a00c78067a4e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18978",
       "triggerID" : "6ab7d4bd2696988e3784291879d1f713d6a9c98d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18997",
       "triggerID" : "4ade37c10c908c0422915aaa489208e6ee62bb0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19027",
       "triggerID" : "57c1b843608a9b63d143ead5dd5168613bb13969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226",
       "triggerID" : "803df61d0d04f7e7403d1177325a365e9bbafab5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461",
       "triggerID" : "2288afba514c322841e5479a7fc7f6cdc8bf8b2d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "613a47bc53981cd9bc4c1fe87c30713fa0f5f2be",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8f2dc4ec3e26f1908ae5d15f194bf70ca7dab27e UNKNOWN
   * 803df61d0d04f7e7403d1177325a365e9bbafab5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19226) 
   * 2288afba514c322841e5479a7fc7f6cdc8bf8b2d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19461) 
   * 613a47bc53981cd9bc4c1fe87c30713fa0f5f2be UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1305083315


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/ActiveAction.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1696112913

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d688280b7fde224d75843893290db559bcc169f5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19480",
       "triggerID" : "d688280b7fde224d75843893290db559bcc169f5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d688280b7fde224d75843893290db559bcc169f5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19480) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 merged pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #9209:
URL: https://github.com/apache/hudi/pull/9209


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6539] New LSM tree style archived timeline [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1393515340


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -164,230 +101,32 @@ public boolean archiveIfRequired(HoodieEngineContext context, boolean acquireLoc
         // there is no owner or instant time per se for archival.
         txnManager.beginTransaction(Option.empty(), Option.empty());
       }
-      List<HoodieInstant> instantsToArchive = getInstantsToArchive().collect(Collectors.toList());
-      verifyLastMergeArchiveFilesIfNecessary(context);
+      // Sort again because the cleaning and rollback instants could break the sequence.
+      List<ActiveAction> instantsToArchive = getInstantsToArchive().sorted().collect(Collectors.toList());
       boolean success = true;
       if (!instantsToArchive.isEmpty()) {
-        this.writer = openWriter();
         LOG.info("Archiving instants " + instantsToArchive);
-        archive(context, instantsToArchive);
+        Consumer<Exception> exceptionHandler = e -> {
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw new HoodieException(e);
+          }
+        };
+        this.timelineWriter.write(instantsToArchive, Option.of(action -> deleteAnyLeftOverMarkers(context, action)), Option.of(exceptionHandler));
         LOG.info("Deleting archived instants " + instantsToArchive);
         success = deleteArchivedInstants(instantsToArchive, context);

Review Comment:
   we should, but currently the redundants left by failed deletion do not affect the correctness, the completion time would be still loaded correctly if an instant locates at both avtive and archived timelines.
   
   We need to think through the design though, it is arduous to maintain the whole multiple handlings as atomic.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/HoodieTimelineArchiver.java:
##########
@@ -164,230 +101,32 @@ public boolean archiveIfRequired(HoodieEngineContext context, boolean acquireLoc
         // there is no owner or instant time per se for archival.
         txnManager.beginTransaction(Option.empty(), Option.empty());
       }
-      List<HoodieInstant> instantsToArchive = getInstantsToArchive().collect(Collectors.toList());
-      verifyLastMergeArchiveFilesIfNecessary(context);
+      // Sort again because the cleaning and rollback instants could break the sequence.
+      List<ActiveAction> instantsToArchive = getInstantsToArchive().sorted().collect(Collectors.toList());
       boolean success = true;
       if (!instantsToArchive.isEmpty()) {
-        this.writer = openWriter();
         LOG.info("Archiving instants " + instantsToArchive);
-        archive(context, instantsToArchive);
+        Consumer<Exception> exceptionHandler = e -> {
+          if (this.config.isFailOnTimelineArchivingEnabled()) {
+            throw new HoodieException(e);
+          }
+        };
+        this.timelineWriter.write(instantsToArchive, Option.of(action -> deleteAnyLeftOverMarkers(context, action)), Option.of(exceptionHandler));
         LOG.info("Deleting archived instants " + instantsToArchive);
         success = deleteArchivedInstants(instantsToArchive, context);

Review Comment:
   We should, but currently the redundants left by failed deletion do not affect the correctness, the completion time would be still loaded correctly if an instant locates at both avtive and archived timelines.
   
   We need to think through the design though, it is arduous to maintain the whole multiple handlings as atomic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1280276081


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java:
##########
@@ -18,75 +18,125 @@
 
 package org.apache.hudi.common.table.timeline;
 
-import org.apache.hudi.avro.HoodieAvroUtils;
-import org.apache.hudi.avro.model.HoodieArchivedMetaEntry;
-import org.apache.hudi.avro.model.HoodieMergeArchiveFilePlan;
-import org.apache.hudi.common.fs.HoodieWrapperFileSystem;
-import org.apache.hudi.common.model.HoodieLogFile;
-import org.apache.hudi.common.model.HoodiePartitionMetadata;
-import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.avro.model.HoodieArchivedInstant;
 import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
-import org.apache.hudi.common.table.log.HoodieLogFormat;
-import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.common.util.ArchivedInstantReadSchemas;
 import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.FileIOUtils;
 import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
 
+import org.apache.avro.Schema;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.generic.IndexedRecord;
 import org.apache.hadoop.fs.FileStatus;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.nio.charset.StandardCharsets;
+import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.Spliterator;
-import java.util.Spliterators;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.function.Function;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import java.util.stream.StreamSupport;
+import java.util.stream.Collectors;
 
 /**
- * Represents the Archived Timeline for the Hoodie table. Instants for the last 12 hours (configurable) is in the
- * ActiveTimeline and the rest are in ArchivedTimeline.
- * <p>
- * </p>
- * Instants are read from the archive file during initialization and never refreshed. To refresh, clients need to call
- * reload()
- * <p>
- * </p>
- * This class can be serialized and de-serialized and on de-serialization the FileSystem is re-initialized.
+ * Represents the Archived Timeline for the Hoodie table.
+ *
+ * <p>After several instants are accumulated as a batch on the active timeline, they would be archived as a parquet file into the archived timeline.
+ * In general the archived timeline is comprised with parquet files with LSM style file layout. Each new operation to the archived timeline generates
+ * a new snapshot version. Theoretically, there could be multiple snapshot versions on the archived timeline.
+ *
+ * <p><h2>The Archived Timeline Layout</h2>
+ *
+ * <pre>
+ *   t111, t112 ... t120 ... ->
+ *   \              /
+ *      \        /
+ *          |
+ *          V
+ *   t111_t120_0.parquet, t101_t110_0.parquet,...  t11_t20_0.parquet    L0
+ *                                  \                    /
+ *                                     \              /
+ *                                            |
+ *                                            V
+ *                                    t11_t100_1.parquet                L1
+ *
+ *      manifest_1, manifest_2, ... manifest_12
+ *          |           |               |
+ *          V           V               V
+ *      _version_1, _version_2, ... _version_12
+ * </pre>
+ *
+ * <p><h2>The LSM Tree Compaction</h2>
+ * Use the universal compaction strategy, that is: when N(by default 10) number of parquet files exist in the current layer, they are merged and flush as a large file in the next layer.
+ * We have no limit for the layer number, assumes there are 10 instants for each file in L0, there could be 100 instants per-file in L1,
+ * so 3000 instants could be represented as 3 parquets in L2, it is pretty fast if we use concurrent read.
+ *
+ * <p>The benchmark shows 1000 instants read cost about 10 ms.
+ *
+ * <p><h2>The Archiver & Reader Snapshot Isolation</h2>
+ *
+ * <p>In order to make snapshot isolation of the archived timeline write/read, we add two kinds of metadata files for the LSM tree version management:
+ * <ol>
+ *   <li>Manifest file: Each new file in layer 0 or each compaction would generate a new manifest file, the manifest file records the valid file handles of the latest snapshot;</li>
+ *   <li>Version file: A version file is generated right after a complete manifest file is formed.</li>
+ * </ol>
+ *
+ * <p><h2>The Reader Workflow</h2>
+ * <ul>
+ *   <li>read the latest version;</li>
+ *   <li>read the manifest file for valid file handles;</li>
+ *   <li>probably do a data skipping with the parquet file name max min timestamp.</li>
+ * </ul>
+ *
+ * <p><h2>The Legacy Files Cleaning and Read Retention</h2>
+ * Only triggers file cleaning after a valid compaction.
+ *
+ * <p><h3>Clean Strategy</h3></p>

Review Comment:
   Either is okay, the version number based cleaning works better when the timeline is committed more frequently, because we do not need timetravel queries on the timeline, the cleaning can be more radical.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9209:
URL: https://github.com/apache/hudi/pull/9209#discussion_r1287849911


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/ArchivedTimelineReadBenchmark.scala:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.DummyActiveInstant
+import org.apache.hudi.client.common.HoodieJavaEngineContext
+import org.apache.hudi.client.utils.{ActiveInstant, ArchivedTimelineWriter}
+import org.apache.hudi.common.model.{HoodieAvroPayload, HoodieTableType, WriteOperationType}
+import org.apache.hudi.common.table.timeline.{HoodieArchivedTimeline, HoodieInstant}
+import org.apache.hudi.common.testutils.{HoodieTestTable, HoodieTestUtils}
+import org.apache.hudi.config.{HoodieIndexConfig, HoodieWriteConfig}
+import org.apache.hudi.index.HoodieIndex.IndexType
+import org.apache.hudi.table.HoodieJavaTable
+import org.apache.spark.hudi.benchmark.{HoodieBenchmark, HoodieBenchmarkBase}
+
+import java.util
+import scala.collection.JavaConverters._
+
+object ArchivedTimelineReadBenchmark extends HoodieBenchmarkBase {

Review Comment:
   yes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1649184386

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62001bbc6c5d9306f95d4bca3dd2bada3ca5c898 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804) 
   * 9889e40cdf17f6f24ddefff010a063d4dd2c58e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18820) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9209: [HUDI-6539] New LSM tree style archived timeline

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9209:
URL: https://github.com/apache/hudi/pull/9209#issuecomment-1649174677

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18799",
       "triggerID" : "a71bc4b8c9eb8a9f0bf61d20934c62a9469c4fd1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804",
       "triggerID" : "62001bbc6c5d9306f95d4bca3dd2bada3ca5c898",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9889e40cdf17f6f24ddefff010a063d4dd2c58e7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62001bbc6c5d9306f95d4bca3dd2bada3ca5c898 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18804) 
   * 9889e40cdf17f6f24ddefff010a063d4dd2c58e7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org