[GitHub] [hudi] prashantwason opened a new pull request #2064: [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason opened a new pull request #2064:
URL: https://github.com/apache/hudi/pull/2064


   ## What is the purpose of the pull request
   
   This is an implementation of HUDI RFC-15.
   
   ## Brief change log
   
   Notable new classes:
    HoodieMetadata - The abstract interface through which the Metadata is accessed
    HoodieMetadataImpl - Implementation of Metadata as per RFC-15
    HoodieMetadataIndex - A HoodieIndex specialized for Metadata to speed up record tagging and improve metadata table performance
    HoodieMetadataPayload - Payload for Metadata Table with custom logic for record merging
    HoodieMetadataFileSystemView - File System View that uses Metadata Table instead of file listings
    metadataSchema.txt - The schema being used for metadata table
   
   ## Verify this pull request
   
   This change added these tests:
    TestHoodieMetadata
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r501943042



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java
##########
@@ -57,41 +57,58 @@
   private static final Logger LOG = LogManager.getLogger(HoodieMergedLogRecordScanner.class);
 
   // Final map of compacted/merged records
-  private final ExternalSpillableMap<String, HoodieRecord<? extends HoodieRecordPayload>> records;
+  protected final ExternalSpillableMap<String, HoodieRecord<? extends HoodieRecordPayload>> records;
 
   // count of merged records in log
   private long numMergedRecordsInLog;
-
+  private long maxMemorySizeInBytes;
   // Stores the total time taken to perform reading and merging of log blocks
-  private final long totalTimeTakenToReadAndMergeBlocks;
+  private long totalTimeTakenToReadAndMergeBlocks;
   // A timer for calculating elapsed time in millis
   public final HoodieTimer timer = new HoodieTimer();
 
   @SuppressWarnings("unchecked")
   public HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<String> logFilePaths, Schema readerSchema,
       String latestInstantTime, Long maxMemorySizeInBytes, boolean readBlocksLazily, boolean reverseReader,
       int bufferSize, String spillableMapBasePath) {
+    this(fs, basePath, logFilePaths, readerSchema, latestInstantTime, maxMemorySizeInBytes, readBlocksLazily,
+        reverseReader, bufferSize, spillableMapBasePath, true);
+  }
+
+  @SuppressWarnings("unchecked")
+  public HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<String> logFilePaths, Schema readerSchema,

Review comment:
       moving as comment under HUDI-1319




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-712708891


   Moved to rfc-15 branch with another PR.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491330366



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -174,8 +174,17 @@ public static String getRelativePartitionPath(Path basePath, Path fullPartitionP
   /**
    * Obtain all the partition paths, that are present in this table, denoted by presence of
    * {@link HoodiePartitionMetadata#HOODIE_PARTITION_METAFILE}.
+   *
+   * If thee basePathStr is a subdirectory of .hoodie folder then we assume that the partitions of an internal
+   * table (a hoodie table within the .hoodie directory) are to be obtained.
+   *
+   * @param fs FileSystem instance
+   * @param basePathStr base directory
    */
   public static List<String> getAllFoldersWithPartitionMetaFile(FileSystem fs, String basePathStr) throws IOException {
+    // If the basePathStr is a folder within the .hoodie directory then we are listing partitions within an
+    // internal table.
+    final boolean isInternalTable = basePathStr.contains(HoodieTableMetaClient.METAFOLDER_NAME);

Review comment:
       I was thinking ahead for other internal tables which will be created in the .hoodie folder. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491311373



##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -755,6 +769,29 @@ public int getBootstrapParallelism() {
     return Integer.parseInt(props.getProperty(HoodieBootstrapConfig.BOOTSTRAP_PARALLELISM));
   }
 
+  /**
+   * File listing metadata configs.
+   */
+  public boolean useFileListingMetadata() {
+    return Boolean.parseBoolean(props.getProperty(USE_FILE_LISTING_METADATA));
+  }
+
+  public boolean getFileListingMetadataVerify() {
+    return Boolean.parseBoolean(props.getProperty(FILE_LISTING_METADATA_VERIFY));
+  }
+
+  public Option<HoodieCompactionConfig> getMetadataCompactionConfig() throws IOException {

Review comment:
       Not a constant. This is user provided config just like for dataset compaction.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [x] 1. Support for rollbacks in MOR Table
   - [x] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [x] 7. Unit test for rollback of partial commits
   - [x] 8. Schema evolution strategy for metadata table
   - [x] 9. Unit test for marker based rollback
   - [x] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   - [ ] 13. How we are going to add new metadata partitions in the background, as writers/cleaner/compactors keep running.
   - [ ] 14. Fix the case when the table is non-partitioned
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r492406923



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());

Review comment:
       we are not filtering for completed compactions right?  in fact, you wont find an instant with `compaction` as action type in compelted, since we make that a commit action eventually. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))

Review comment:
       metaTimeline.getDeltaCommitTimeline().filterCompletedInstants()? instead of writing this by hand

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());

Review comment:
       .firstInstant() instead?

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))

Review comment:
       Correction. I think we will still honor scheduled clean plans. but overall, the point remains the same. we can just reflect completed cleans IMO. The only side effect would be that when cleaning by number of retained versions, it can lead to less aggressive cleaning in rare cases. for cleaning by commits, it should be fine, as the cleaning is decided based on a watermark commit time  and file slices before that are discarded. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)

Review comment:
       this can be rewritten using `findInstantsInRange()`

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))

Review comment:
       See my note on reflecting compaction after its completed. It has some issues for queries. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r501374106



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java
##########
@@ -244,6 +245,10 @@ private HoodieCleanMetadata runClean(HoodieTable<?> table, HoodieInstant cleanIn
         inflightInstant = cleanInstant;
       }
 
+      // Update Metadata Table before even finishing clean. This ensures that an async clean operation in the
+      // background does not lead to stale metadata being returned from Metadata Table.
+      HoodieMetadata.update(config, cleanerPlan, cleanInstant.getTimestamp());

Review comment:
       Created HUDI-1319




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

bvaradar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r495413600



##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+
+    StringBuffer out = new StringBuffer("\n");
+    if (!HoodieMetadata.exists(HoodieCLI.basePath)) {
+      out.append("=== Metadata Table not initilized. Using file listing to get list of partitions. ===\n\n");
+    }
+
+    long t1 = System.currentTimeMillis();
+    List<String> partitions = HoodieMetadata.getAllPartitionPaths(HoodieCLI.fs, HoodieCLI.basePath, false);
+    long t2 = System.currentTimeMillis();
+
+    int[] count = {0};
+    partitions.stream().sorted((p1, p2) -> p2.compareTo(p1)).forEach(p -> {
+      out.append(p);
+      if (++count[0] % 15 == 0) {
+        out.append("\n");
+      } else {
+        out.append(", ");
+      }
+    });
+
+    out.append(String.format("\n\n=== List of partitions retrieved in %.2fsec ===", (t2 - t1) / 1000.0));
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-files", help = "Print a list of all files in a partition from the metadata")
+  public String listFiles(

Review comment:
       @prashantwason : Can you make sure we can query when the underlying table is non-partitioned 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491331857



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * This is a payload which saves information about a single entry in the Metadata Table. The type of the entry is
+ * determined by the "type" saved within the record. The following types of entries are saved:
+ *
+ *   1. List of partitions: There is a single such record
+ *         key="__all_partitions__"
+ *         filenameToSizeMap={"2020/01/01": 0, "2020/01/02": 0, ...}
+ *
+ *   2. List of files in a Partition: There is one such record for each partition
+ *         key=Partition name
+ *         filenameToSizeMap={"file1.parquet": 12345, "file2.parquet": 56789, "file1.log": 9876,
+ *                            "file0.parquet": -1, ...}
+ *
+ *      For deleted files, -1 is used as the size.
+ *
+ *  During compaction on the table, the deletions are merged with additions and hence pruned.
+ */
+public class HoodieMetadataPayload implements HoodieRecordPayload<HoodieMetadataPayload> {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataPayload.class);
+
+  // Represents the size stored for a deleted file
+  private static final long DELETED_FILE_SIZE = -1;
+
+  // Key and type for the metadata record
+  private final String metadataKey;
+  private final PayloadType type;
+
+  // Filenames which are part of this record
+  // key=filename, value=file size (or DELETED_FILE_SIZE to represent a deleted file)
+  private final Map<String, Long> filenameMap = new HashMap<>();
+
+  // Type of the metadata record
+  public enum PayloadType {
+    PARTITION_LIST(1),        // list of partitions
+    PARTITION_FILES(2);       // list of files in a partition
+
+    private final int value;

Review comment:
       That and the fact that I want to avoid 0. In my experience 0 is hard to distinguish from not assigned or corrupted. 
   Without explicit numbers any reordering is hard to catch (also IDEs may reorder).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497079467



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)

Review comment:
       Rewritten using findInstantsAfterOrEquals




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497095419



##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+
+    StringBuffer out = new StringBuffer("\n");
+    if (!HoodieMetadata.exists(HoodieCLI.basePath)) {
+      out.append("=== Metadata Table not initilized. Using file listing to get list of partitions. ===\n\n");
+    }
+
+    long t1 = System.currentTimeMillis();
+    List<String> partitions = HoodieMetadata.getAllPartitionPaths(HoodieCLI.fs, HoodieCLI.basePath, false);
+    long t2 = System.currentTimeMillis();
+
+    int[] count = {0};
+    partitions.stream().sorted((p1, p2) -> p2.compareTo(p1)).forEach(p -> {
+      out.append(p);
+      if (++count[0] % 15 == 0) {
+        out.append("\n");
+      } else {
+        out.append(", ");
+      }
+    });
+
+    out.append(String.format("\n\n=== List of partitions retrieved in %.2fsec ===", (t2 - t1) / 1000.0));
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-files", help = "Print a list of all files in a partition from the metadata")
+  public String listFiles(
+      @CliOption(key = {"partition"}, help = "Name of the partition to list files", mandatory = true)
+      final String partition) throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();

Review comment:
       The MetaCommand code assumes that you will call "metadata init" first. This is to ensure that listFiles can be performed both with and without the metadata being initialized for performance comparison purpose.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-696429882






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497068099



##########
File path: hudi-client/src/main/resources/metadataSchema.txt
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+    "namespace": "hudi.metadata",
+    "type": "record",
+    "name": "metadata",

Review comment:
       Yes, I will take care of this in schema evolution work.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-700996406


   > Also see a lot of this polluting the test logs.  can we remove or avoid them on travis
   
   Filed https://github.com/apache/hudi/pull/2132 for this issue.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [ ] 1. Support for rollbacks in MOR Table
   - [x] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [ ] 7. Unit test for rollback of partial commits
   - [ ] 8. Schema evolution strategy for metadata table
   - [ ] 9. Unit test for marker based rollback
   - [ ] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491315758



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadata.java
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.exception.TableNotFoundException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+/**
+ * Defines the interface through which file listing metadata is created, updated and accessed. This metadata is
+ * saved within an internal Metadata Table.
+ *
+ * If file listing metadata is disabled, the functions default to using listStatus(...) RPC calls to retrieve
+ * file listings from the file system.
+ */
+public class HoodieMetadata {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadata.class);
+
+  // Instances of metadata for each basePath
+  private static Map<String, HoodieMetadataImpl> instances = new HashMap<>();

Review comment:
       Each time you create an instance of this class, it needs to load the timeline and check for sync, etc. So ideally an instance should be created only once. But the HoodieMetadata is being used in multiple classes (HoodieWriteClient and various XXXCommitActionExectors) so I could not  just keep it at one place.
   
   >> Unless we are doing something here across multiple base paths,
   
   Yes, we have a usecase in ETL where the same data is written to multiple HUDI tables (cell / row) in parallel. Hence, we need this to be a map.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-703192383


   @prashantwason there is a `rfc-15` branch under apache now. We can target all our PRs to this branch going forward. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r488962216



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -194,6 +196,35 @@ public boolean archiveIfRequired(JavaSparkContext jsc) throws IOException {
         .collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
             HoodieInstant.getComparableAction(i.getAction()))));
 
+    // If metadata table is enabled, do not archive instants which are more recent that the latest compaction
+    // of the metadata table. This is required for metadata table sync.
+    if (config.useFileListingMetadata()) {

Review comment:
       lets find a way to avoid code duplication here?

##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -194,6 +196,35 @@ public boolean archiveIfRequired(JavaSparkContext jsc) throws IOException {
         .collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
             HoodieInstant.getComparableAction(i.getAction()))));
 
+    // If metadata table is enabled, do not archive instants which are more recent that the latest compaction
+    // of the metadata table. This is required for metadata table sync.
+    if (config.useFileListingMetadata()) {
+      Option<String> latestCompaction = HoodieMetadata.getLatestCompactionTimestamp(config.getBasePath());
+      if (latestCompaction.isPresent()) {
+        LOG.info("Limiting archiving of instants to last compaction on metadata table at " + latestCompaction.get());
+        instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
+            latestCompaction.get()));
+      } else {
+        LOG.info("Not arching instants as there is no compaction yet of the metadata table");

Review comment:
       typo: archiving 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r492405639



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadata.java
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.exception.TableNotFoundException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+/**
+ * Defines the interface through which file listing metadata is created, updated and accessed. This metadata is
+ * saved within an internal Metadata Table.
+ *
+ * If file listing metadata is disabled, the functions default to using listStatus(...) RPC calls to retrieve
+ * file listings from the file system.
+ */
+public class HoodieMetadata {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadata.class);
+
+  // Instances of metadata for each basePath
+  private static Map<String, HoodieMetadataImpl> instances = new HashMap<>();

Review comment:
       I don't know if sharing this object leads to any real perf gains. for the use-case you mention, those are two different hudi tables anyway right.  
   
   yes, we need to pass this around the layers. best way is to stick this into `HoodieTable` just like it has a member variable for `index`. Then you can easily access this across layers. This also has to be made `Serializable`. In general, global static objects, when clearly there is a scope for separating the instances is kind of an anti pattern in Java land. Its okay if you want to defer this to me. Just saying that we will probably have to change this code eventually for these reasons. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());

Review comment:
       we are not filtering for completed compactions right?  in fact, you wont find an instant with `compaction` as action type in compelted, since we make that a commit action eventually. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))

Review comment:
       metaTimeline.getDeltaCommitTimeline().filterCompletedInstants()? instead of writing this by hand

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());

Review comment:
       .firstInstant() instead?

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))

Review comment:
       Correction. I think we will still honor scheduled clean plans. but overall, the point remains the same. we can just reflect completed cleans IMO. The only side effect would be that when cleaning by number of retained versions, it can lead to less aggressive cleaning in rare cases. for cleaning by commits, it should be fine, as the cleaning is decided based on a watermark commit time  and file slices before that are discarded. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)

Review comment:
       this can be rewritten using `findInstantsInRange()`

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))

Review comment:
       See my note on reflecting compaction after its completed. It has some issues for queries. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r490967091



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -194,6 +196,35 @@ public boolean archiveIfRequired(JavaSparkContext jsc) throws IOException {
         .collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
             HoodieInstant.getComparableAction(i.getAction()))));
 
+    // If metadata table is enabled, do not archive instants which are more recent that the latest compaction
+    // of the metadata table. This is required for metadata table sync.
+    if (config.useFileListingMetadata()) {
+      Option<String> latestCompaction = HoodieMetadata.getLatestCompactionTimestamp(config.getBasePath());
+      if (latestCompaction.isPresent()) {
+        LOG.info("Limiting archiving of instants to last compaction on metadata table at " + latestCompaction.get());
+        instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
+            latestCompaction.get()));
+      } else {
+        LOG.info("Not arching instants as there is no compaction yet of the metadata table");
+        instants = instants.filter(i -> false);
+      }
+    }
+
+    // For metadata tables, ensure commits >= latest compaction commit are retained. This is required for
+    // metadata table sync.
+    if (HoodieMetadata.isMetadataTable(config.getBasePath())) {
+      Option<HoodieInstant> latestCompactionInstant =
+          table.getActiveTimeline().filterPendingCompactionTimeline().lastInstant();
+      if (latestCompactionInstant.isPresent()) {
+        LOG.info("Limiting archiving of instants on metadata table to last compaction at " + latestCompactionInstant.get());
+        instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
+            latestCompactionInstant.get().getTimestamp()));
+      } else {
+        LOG.info("Not archiving instants on metdata table as there is no compaction yet");
+        instants = instants.filter(i -> false);

Review comment:
       same here

##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -194,6 +196,35 @@ public boolean archiveIfRequired(JavaSparkContext jsc) throws IOException {
         .collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
             HoodieInstant.getComparableAction(i.getAction()))));
 
+    // If metadata table is enabled, do not archive instants which are more recent that the latest compaction
+    // of the metadata table. This is required for metadata table sync.
+    if (config.useFileListingMetadata()) {
+      Option<String> latestCompaction = HoodieMetadata.getLatestCompactionTimestamp(config.getBasePath());
+      if (latestCompaction.isPresent()) {
+        LOG.info("Limiting archiving of instants to last compaction on metadata table at " + latestCompaction.get());
+        instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
+            latestCompaction.get()));
+      } else {
+        LOG.info("Not arching instants as there is no compaction yet of the metadata table");
+        instants = instants.filter(i -> false);

Review comment:
       lets use Stream.empty() instead? 

##########
File path: hudi-client/src/main/resources/metadataSchema.txt
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+    "namespace": "hudi.metadata",
+    "type": "record",
+    "name": "metadata",
+    "fields": [
+        {
+            "name": "key",
+            "type": "string"
+        },
+        {
+            "name": "type",
+            "type": "int",
+            "doc": "Type of the metadata record (refer to HoodieMetadataPayload)"
+        },
+        {   "name": "filenameToSizeMap",
+            "type": {
+                "type": "map",
+                "doc": "Filenames mapped to their sizes",
+                "values": {
+                    "type": "long",
+                    "doc": "Size of this file in bytes or -1 for deleted files"

Review comment:
       I suggest using an explicit boolean for denoting deletion. Also wondering if we should have a record to encapsulate what we track for each file. today its only size. but I can imagine us tracking things like num records to re-implement our file sizing etc efficiently .

##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -174,8 +174,17 @@ public static String getRelativePartitionPath(Path basePath, Path fullPartitionP
   /**
    * Obtain all the partition paths, that are present in this table, denoted by presence of
    * {@link HoodiePartitionMetadata#HOODIE_PARTITION_METAFILE}.
+   *
+   * If thee basePathStr is a subdirectory of .hoodie folder then we assume that the partitions of an internal
+   * table (a hoodie table within the .hoodie directory) are to be obtained.
+   *
+   * @param fs FileSystem instance
+   * @param basePathStr base directory
    */
   public static List<String> getAllFoldersWithPartitionMetaFile(FileSystem fs, String basePathStr) throws IOException {
+    // If the basePathStr is a folder within the .hoodie directory then we are listing partitions within an
+    // internal table.
+    final boolean isInternalTable = basePathStr.contains(HoodieTableMetaClient.METAFOLDER_NAME);

Review comment:
       rename:isMetadataTable?  We can use one term consistently 

##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java
##########
@@ -57,41 +57,58 @@
   private static final Logger LOG = LogManager.getLogger(HoodieMergedLogRecordScanner.class);
 
   // Final map of compacted/merged records
-  private final ExternalSpillableMap<String, HoodieRecord<? extends HoodieRecordPayload>> records;
+  protected final ExternalSpillableMap<String, HoodieRecord<? extends HoodieRecordPayload>> records;
 
   // count of merged records in log
   private long numMergedRecordsInLog;
-
+  private long maxMemorySizeInBytes;
   // Stores the total time taken to perform reading and merging of log blocks
-  private final long totalTimeTakenToReadAndMergeBlocks;
+  private long totalTimeTakenToReadAndMergeBlocks;
   // A timer for calculating elapsed time in millis
   public final HoodieTimer timer = new HoodieTimer();
 
   @SuppressWarnings("unchecked")
   public HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<String> logFilePaths, Schema readerSchema,
       String latestInstantTime, Long maxMemorySizeInBytes, boolean readBlocksLazily, boolean reverseReader,
       int bufferSize, String spillableMapBasePath) {
+    this(fs, basePath, logFilePaths, readerSchema, latestInstantTime, maxMemorySizeInBytes, readBlocksLazily,
+        reverseReader, bufferSize, spillableMapBasePath, true);
+  }
+
+  @SuppressWarnings("unchecked")
+  public HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<String> logFilePaths, Schema readerSchema,

Review comment:
       General comment on concurrent writes to the log. We can have situation where the logged blocks are not in increasing instant time order.. We should ensure we read all the headers first and then process them consistently in instant time order. ? We need some tests around such scenarios. 

##########
File path: hudi-client/src/main/resources/metadataSchema.txt
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+    "namespace": "hudi.metadata",
+    "type": "record",
+    "name": "metadata",

Review comment:
       if we will eventually add different partitions with other schema. does it make sense to pull this into a record like `FileSystemMetadata` , keep it optional. and later we can extend with other fields like `FileGroupStatsMetadata` , `FileGroupBloomFilterMetadata` etc.? 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");

Review comment:
       nice! 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";

Review comment:
       rename : `_all_partition_paths` to be consistent with rest of code. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);

Review comment:
       I think I also posted this as a general design question. Repeating here. how does this interplay with already in-flight compactions/cleaning. that may not have metadata table enabled? 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero

Review comment:
       how about Math.max(1, Math.min(...))?

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)
+            && (metaSyncTimestamp.isEmpty() || HoodieTimeline.compareTimestamps(i.getTimestamp(),
+                HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp)))
+        .collect(Collectors.toMap(HoodieInstant::getTimestamp, Function.identity()));
+
+    List<HoodieInstant> instantsToSync = new LinkedList<>();
+    datasetInstants.forEach(instant -> {
+      if (metadataInstantMap.containsKey(instant.getTimestamp())) {
+        // instant already synced to metadata table
+        if (!instantsToSync.isEmpty()) {
+          LOG.warn("Found out-of-order already synced instant " + instant + ". Instants to sync=" + instantsToSync);
+        }
+      } else {
+        instantsToSync.add(instant);
+      }
+    });
+
+    // TODO: async clean and async compaction are not handled here. They have a timestamp which is in the past

Review comment:
       these are pretty important to handle. I also left a comment above around this. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)
+            && (metaSyncTimestamp.isEmpty() || HoodieTimeline.compareTimestamps(i.getTimestamp(),
+                HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp)))
+        .collect(Collectors.toMap(HoodieInstant::getTimestamp, Function.identity()));
+
+    List<HoodieInstant> instantsToSync = new LinkedList<>();
+    datasetInstants.forEach(instant -> {
+      if (metadataInstantMap.containsKey(instant.getTimestamp())) {
+        // instant already synced to metadata table
+        if (!instantsToSync.isEmpty()) {
+          LOG.warn("Found out-of-order already synced instant " + instant + ". Instants to sync=" + instantsToSync);
+        }
+      } else {
+        instantsToSync.add(instant);
+      }
+    });
+
+    // TODO: async clean and async compaction are not handled here. They have a timestamp which is in the past
+    // (when the operation was scheduled) and even on completion they retain their old timestamp.
+    return instantsToSync;
+  }
+
+  /**
+   * Update from {@code HoodieCommitMetadata}.
+   *
+   * @param commitMetadata {@code HoodieCommitMetadata}
+   * @param instantTime Timestamp at which the commit was performed
+   */
+  void update(HoodieCommitMetadata commitMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    commitMetadata.getPartitionToWriteStats().forEach((partition, writeStats) -> {
+      Map<String, Long> newFiles = new HashMap<>(writeStats.size());
+      writeStats.forEach(hoodieWriteStat -> {
+        String pathWithPartition = hoodieWriteStat.getPath();
+        if (pathWithPartition == null) {
+          // Empty partition
+          return;
+        }
+        String filename = pathWithPartition.substring(partition.length() + 1);
+        ValidationUtils.checkState(!newFiles.containsKey(filename), "Duplicate files in HoodieCommitMetadata");
+        newFiles.put(filename, hoodieWriteStat.getTotalWriteBytes());
+      });
+
+      // New files added to a partition
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.of(newFiles),
+          Option.empty());
+      records.add(record);
+    });
+
+    // New partitions created
+    HoodieRecord record = HoodieMetadataPayload.createPartitionListRecord(
+        new ArrayList<String>(commitMetadata.getPartitionToWriteStats().keySet()));
+    records.add(record);
+
+    LOG.info("Updating at " + instantTime + " from Commit/" + commitMetadata.getOperationType()
+        + ". #partitions_updated=" + records.size());
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanerPlan}.
+   *
+   * @param cleanerPlan {@code HoodieCleanerPlan}
+   * @param instantTime Timestamp at which the clean plan was generated
+   */
+  void update(HoodieCleanerPlan cleanerPlan, String instantTime) {
+    // TODO: Cleaner update is called even before the operation has completed. Hence, this function may be called
+    // multiple times and we dont need to update metadata again.
+    HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+    long cnt = timeline.filterCompletedInstants().getInstants().filter(i -> i.getTimestamp().equals(instantTime)).count();
+    if (cnt == 1) {
+      LOG.info("Ignoring update from cleaner plan for already completed instant " + instantTime);
+      return;
+    }
+
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+    cleanerPlan.getFilePathsToBeDeletedPerPartition().forEach((partition, deletedPathInfo) -> {
+      fileDeleteCount[0] += deletedPathInfo.size();
+
+      // Files deleted from a partition
+      List<String> deletedFilenames = deletedPathInfo.stream().map(p -> new Path(p.getFilePath()).getName())
+          .collect(Collectors.toList());
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(deletedFilenames));
+      records.add(record);
+    });
+
+    LOG.info("Updating at " + instantTime + " from CleanerPlan. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanMetadata}.
+   *
+   * @param cleanMetadata {@code HoodieCleanMetadata}
+   * @param instantTime Timestamp at which the clean was completed
+   */
+  private void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+
+    cleanMetadata.getPartitionMetadata().forEach((partition, partitionMetadata) -> {
+      // Files deleted from a partition
+      List<String> deletedFiles = partitionMetadata.getSuccessDeleteFiles();
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(new ArrayList<>(deletedFiles)));
+
+      records.add(record);
+      fileDeleteCount[0] += deletedFiles.size();
+    });
+
+    LOG.info("Updating at " + instantTime + " from Clean. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieRestoreMetadata}.
+   *
+   * @param restoreMetadata {@code HoodieRestoreMetadata}
+   * @param instantTime Timestamp at which the restore was performed
+   */
+  void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    restoreMetadata.getHoodieRestoreMetadata().values().forEach(rms -> {
+      rms.forEach(rm -> processRollbackMetadata(rm, partitionToDeletedFiles));
+    });
+
+    commitRollbackDeletes(partitionToDeletedFiles, instantTime, "Restore");
+  }
+
+  /**
+   * Update from {@code HoodieRollbackMetadata}.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param instantTime Timestamp at which the rollback was performed
+   */
+  void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    processRollbackMetadata(rollbackMetadata, partitionToDeletedFiles);
+    commitRollbackDeletes(partitionToDeletedFiles, instantTime, "Rollback");
+  }
+
+  /**
+   * Retrives the partition and deleted files from {@code HoodieRollbackMetadata}.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param partitionToDeletedFiles The {@code Map} to fill with partition and deleted files.
+   */
+  private void processRollbackMetadata(HoodieRollbackMetadata rollbackMetadata,
+                                       Map<String, List<String>> partitionToDeletedFiles) {
+    rollbackMetadata.getPartitionMetadata().values().forEach(pm -> {
+      // Extract deleted file name from the absolute paths saved in getSuccessDeleteFiles()
+      List<String> deletedFiles = pm.getSuccessDeleteFiles().stream().map(p -> new Path(p).getName())
+          .collect(Collectors.toList());
+      if (partitionToDeletedFiles.containsKey(pm.getPartitionPath())) {
+        partitionToDeletedFiles.get(pm.getPartitionPath()).addAll(deletedFiles);
+      } else {
+        partitionToDeletedFiles.put(pm.getPartitionPath(), new ArrayList<>(deletedFiles));
+      }
+    });
+  }
+
+  /**
+   * Create file delete records and commit.
+   *
+   * @param partitionToDeletedFiles {@code Map} of partitions and the deleted files
+   * @param instantTime Timestamp at which the deletes took place
+   * @param operation Type of the operation which caused the files to be deleted
+   */
+  private void commitRollbackDeletes(Map<String, List<String>> partitionToDeletedFiles, String instantTime,
+                                     String operation) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+    partitionToDeletedFiles.forEach((partition, deletedFiles) -> {
+      // Rollbacks deletes instants from timeline. The instant being rolled-back may not have been synced to the
+      // metadata table. Hence, the deleted filed need to be checked against the metadata.
+      try {
+        int origCount = deletedFiles.size();
+        FileStatus[] existingStatuses = getAllFilesInPartition(new Path(basePath, partition));
+        Set<String> currentFiles =
+            Arrays.stream(existingStatuses).map(s -> s.getPath().getName()).collect(Collectors.toSet());
+        deletedFiles.removeIf(f -> !currentFiles.contains(f));
+        if (deletedFiles.size() != origCount) {
+          LOG.warn("Some Files to be deleted as part of " + operation + " at " + instantTime + " were not found in the "
+              + " metadata for partition " + partition
+              + ". To delete = " + origCount + ", found=" + deletedFiles.size());
+        }
+
+        // Commit this record even if there are no deleted files. This will create a delta-commit corresponding to
+        // the instant being synced.
+        fileDeleteCount[0] += deletedFiles.size();
+
+        // Files deleted from a partition
+        HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+            Option.of(new ArrayList<>(deletedFiles)));
+        records.add(record);
+
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to commit rollback deletes at instant " + instantTime, e);
+      }
+    });
+
+    if (!records.isEmpty()) {
+      LOG.info("Updating at " + instantTime + " from " + operation + ". #partitions_updated=" + records.size()
+          + ", #files_deleted=" + fileDeleteCount);
+      commit(records, instantTime, true);
+    }
+  }
+
+  /**
+   * Commit the {@code HoodieRecord}s to Metadata Table as a new delta-commit.
+   *
+   * @param records The records to commit
+   * @param instantTime The timestamp of instant to create
+   * @param performUpdate If True, records are upserted. If False, records are inserted
+   */
+  private synchronized void commit(List<HoodieRecord> records, String instantTime, boolean performUpdate) {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be committed in readonly mode");
+    ValidationUtils.checkArgument(!records.isEmpty());
+
+    // Close all readers
+    if (basefileReader != null) {
+      basefileReader.close();
+      basefileReader = null;
+    }
+    logRecordScanner = null;
+
+    try (HoodieWriteClient writeClient = new HoodieWriteClient(jsc, config, true)) {
+      writeClient.startCommitWithTime(instantTime);
+      List<WriteStatus> statuses;
+      if (performUpdate) {
+        statuses = writeClient.upsert(jsc.parallelize(records), instantTime).collect();
+      } else {
+        statuses = writeClient.insert(jsc.parallelize(records), instantTime).collect();
+      }
+      statuses.forEach(writeStatus -> {
+        if (writeStatus.hasErrors()) {
+          throw new HoodieMetadataException("Failed to commit metadata table records at instant " + instantTime);
+        }
+      });
+    }
+  }
+
+  static String getMetadataTableBasePath(String tableBasePath) {
+    return tableBasePath + Path.SEPARATOR + METADATA_TABLE_REL_PATH;
+  }
+
+  /**
+   * Returns a list of all partitions.
+   */
+  List<String> getAllPartitionPaths() throws IOException {
+    final Timer.Context context = this.metrics.getMetadataCtx(LOOKUP_PARTITIONS_STR);
+
+    Option<HoodieRecord<HoodieMetadataPayload>> hoodieRecord = getMergedRecordByKey(HoodieMetadataImpl.RECORDKEY_PARTITION_LIST);
+    List<String> partitions = Collections.emptyList();
+    if (hoodieRecord.isPresent()) {
+      if (!hoodieRecord.get().getData().getDeletions().isEmpty()) {
+        throw new HoodieMetadataException("Metadata partition list record is inconsistent: "
+            + hoodieRecord.get().getData());
+      }
+
+      partitions = hoodieRecord.get().getData().getFilenames();
+    }
+
+    ++numLookupsAllPartitions;
+
+    long validateTime = 0;
+    if (validateLookups) {
+      // Validate the Metadata Table data by listing the partitions from the file system
+      final Timer.Context contextValidate = this.metrics.getMetadataCtx(VALIDATE_PARTITIONS_STR);
+      List<String> actualPartitions  = FSUtils.getAllPartitionPaths(metaClient.getFs(), basePath, false);
+
+      Collections.sort(actualPartitions);
+      Collections.sort(partitions);
+      if (!actualPartitions.equals(partitions)) {
+        LOG.error("Validation of metadata partition list failed. Lists do not match.");
+        LOG.error("Partitions from metadata: " + Arrays.toString(partitions.toArray()));
+        LOG.error("Partitions from file system: " + Arrays.toString(actualPartitions.toArray()));
+
+        ++numValidationErrors;
+        metrics.updateMetadataMetrics(VALIDATE_ERRORS_STR, 0, numValidationErrors);
+      }
+
+      // Return the direct listing as it should be correct
+      partitions = actualPartitions;
+
+      if (contextValidate != null) {
+        validateTime = metrics.getDurationInMs(contextValidate.stop());
+        metrics.updateMetadataMetrics(VALIDATE_PARTITIONS_STR, validateTime, numLookupsAllPartitions);
+      }
+    }
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop()) - validateTime;
+      metrics.updateMetadataMetrics(LOOKUP_PARTITIONS_STR, durationInMs, numLookupsAllPartitions);
+    }
+
+    return partitions;
+  }
+
+  /**
+   * Return all the files from the partition.
+   *
+   * @param partitionPath The absolute path of the partition
+   */
+  FileStatus[] getAllFilesInPartition(Path partitionPath) throws IOException {
+    final Timer.Context context = this.metrics.getMetadataCtx(LOOKUP_FILES_STR);
+
+    String partitionName = FSUtils.getRelativePartitionPath(new Path(basePath), partitionPath);
+
+    Option<HoodieRecord<HoodieMetadataPayload>> hoodieRecord = getMergedRecordByKey(partitionName);
+    FileStatus[] statuses = {};
+    if (hoodieRecord.isPresent()) {
+      if (!hoodieRecord.get().getData().getDeletions().isEmpty()) {
+        throw new HoodieMetadataException("Metadata record for partition " + partitionName + " is inconsistent: "
+              + hoodieRecord.get().getData());
+      }
+      statuses = hoodieRecord.get().getData().getFileStatuses(partitionPath);
+    }
+
+    ++numLookupsFilesInPartition;
+
+    long validateTime = 0;
+    if (validateLookups) {
+      // Validate the Metadata Table data by listing the partitions from the file system
+      final Timer.Context contextValidate = this.metrics.getMetadataCtx(VALIDATE_FILES_STR);
+      validateTime = System.currentTimeMillis();
+      // Compare the statuses from metadata to those from listing directly from the file system.
+      // Ignore partition metadata file
+      FileStatus[] directStatuses = metaClient.getFs().listStatus(partitionPath,
+          p -> !p.getName().equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE));
+      List<String> directFilenames = Arrays.stream(directStatuses)
+          .map(s -> s.getPath().getName()).collect(Collectors.toList());
+      List<String> metadataFilenames = Arrays.stream(statuses)
+          .map(s -> s.getPath().getName()).collect(Collectors.toList());
+
+      Collections.sort(metadataFilenames);
+      Collections.sort(directFilenames);
+      if (!metadataFilenames.equals(directFilenames)) {
+        LOG.error("Validation of metadata file listing for partition " + partitionName + " failed.");
+        LOG.error("File list from metadata: " + Arrays.toString(metadataFilenames.toArray()));
+        LOG.error("File list from direct listing: " + Arrays.toString(directFilenames.toArray()));
+
+        ++numValidationErrors;
+        metrics.updateMetadataMetrics(VALIDATE_ERRORS_STR, 0, numValidationErrors);
+      }
+
+      // Return the direct listing as it should be correct
+      statuses = directStatuses;
+
+      if (contextValidate != null) {
+        validateTime = metrics.getDurationInMs(contextValidate.stop());
+        metrics.updateMetadataMetrics(VALIDATE_FILES_STR, validateTime, numLookupsFilesInPartition);
+      }
+    }
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop()) - validateTime;
+      metrics.updateMetadataMetrics(LOOKUP_FILES_STR, durationInMs, numLookupsFilesInPartition);
+    }
+
+    return statuses;
+  }
+
+  /**
+   * Open readers to the base and log files.
+   */
+  private synchronized void openBaseAndLogFiles() throws IOException {
+    if (logRecordScanner != null) {
+      // Already opened
+      return;
+    }
+
+    final Timer.Context context = this.metrics.getMetadataCtx(SCAN_STR);
+
+    // Find the latest file slice
+    HoodieTimeline timeline = metaClient.reloadActiveTimeline();
+    HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+    List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+    ValidationUtils.checkArgument(latestSlices.size() == 1);
+
+    // If the base file is present then create a reader
+    Option<HoodieBaseFile> basefile = latestSlices.get(0).getBaseFile();
+    if (basefile.isPresent()) {
+      String basefilePath = basefile.get().getPath();
+      basefileReader = HoodieFileReaderFactory.getFileReader(hadoopConf, new Path(basefilePath));
+
+      LOG.info("Opened metadata base file from " + basefilePath + " at instant " + basefile.get().getCommitTime());
+    }
+
+    // Open the log record scanner using the log files from the latest file slice
+    List<String> logFilePaths = latestSlices.get(0).getLogFiles().map(o -> o.getPath().toString())
+        .collect(Collectors.toList());
+    Option<HoodieInstant> lastInstant = timeline.filterCompletedInstants().lastInstant();
+    String latestTimestamp = lastInstant.isPresent() ? lastInstant.get().getTimestamp() : "";
+
+    logRecordScanner =
+        new HoodieMetadataMergedLogRecordScanner(FSUtils.getFs(basePath, hadoopConf), metadataBasePath,
+            logFilePaths, schema, latestTimestamp, maxMemorySizeInBytes, bufferSize,
+            config.getSpillableMapBasePath(), null);
+
+    LOG.info("Opened metadata log files from " + logFilePaths + " at instant " + lastInstant);
+
+    ++numScans;
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      metrics.updateMetadataMetrics(SCAN_STR, durationInMs, numScans);
+    }
+  }
+
+  /**
+   * Retrieve the merged {@code HoodieRecord} mapped to the given key.
+   *
+   * @param key The key of the record
+   */
+  private Option<HoodieRecord<HoodieMetadataPayload>> getMergedRecordByKey(String key) throws IOException {
+    openBaseAndLogFiles();
+
+    // Retrieve record from base file
+    HoodieRecord<HoodieMetadataPayload> hoodieRecord = null;
+    if (basefileReader != null) {
+      Option<GenericRecord> baseRecord = basefileReader.getRecordByKey(key);

Review comment:
       note to self: typical use-case is to list all partition paths and then for each pp, find all the file slices etc. so probably merging the base and log in each such call is okay? with inlined hFIle logs in log. this can be much faster. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataFileSystemView.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * {@code HoodieTableFileSystemView} implementation that retrieved partition listings from the Metadata Table.
+ */
+public class HoodieMetadataFileSystemView extends HoodieTableFileSystemView {
+
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataFileSystemView.class);
+
+  public HoodieMetadataFileSystemView(HoodieTableMetaClient metaClient, HoodieTimeline visibleActiveTimeline,
+      boolean enableIncrementalTimelineSync) {
+    super(metaClient, visibleActiveTimeline, enableIncrementalTimelineSync);
+  }
+
+  /**
+   * Return all the files in the partition by reading from the Metadata Table.
+   *
+   * @param partitionPath The absolute path of the partition
+   * @throws IOException
+   */
+  @Override
+  protected FileStatus[] listPartition(Path partitionPath) throws IOException {

Review comment:
       are we still relying on `FSUtils.getAllPartitionPaths()` for obtaining the partition paths. I think we can also bring those usages under HoodieTableFileSystemView and add a `List<String> allPartitionPaths()` to HoodieTableFileSystemView and implement that as well using the metadata table?

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadata.java
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.exception.TableNotFoundException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+/**
+ * Defines the interface through which file listing metadata is created, updated and accessed. This metadata is
+ * saved within an internal Metadata Table.
+ *
+ * If file listing metadata is disabled, the functions default to using listStatus(...) RPC calls to retrieve
+ * file listings from the file system.
+ */
+public class HoodieMetadata {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadata.class);
+
+  // Instances of metadata for each basePath
+  private static Map<String, HoodieMetadataImpl> instances = new HashMap<>();
+
+  /**
+   * Initialize the Metadata Table.
+   *  - If the table does not exist it is created for the first time.
+   *  - If the table exists, it is synced with the instants on the dataset.
+   *
+   * @param jsc {@code JavaSparkContext}
+   * @param writeConfig {@code HoodieWriteConfig} for the writer from which relevant configuration is inherited
+   */
+  public static synchronized void init(JavaSparkContext jsc, HoodieWriteConfig writeConfig) {
+    try {
+      final String basePath = writeConfig.getBasePath();
+      if (instances.containsKey(basePath)) {
+        LOG.warn("Duplicate initialization of metadata for basePath " + basePath);
+      }
+      if (writeConfig.useFileListingMetadata()) {
+        instances.put(basePath, new HoodieMetadataImpl(jsc, jsc.hadoopConfiguration(), basePath, writeConfig, false));
+      } else {
+        if (!isMetadataTable(writeConfig.getBasePath())) {
+          LOG.info("Metadata table is disabled in write config.");
+        }
+      }
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to initialize metadata table", e);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table in read-only mode.
+   *
+   * If the table does not exist, a warning is logged and RPC calls are used to retrieve file listings
+   * from the file system. No updates are applied to the table and it is not synced.
+   *
+   * @param hadoopConf {@code Configuration}
+   * @param basePath The basePath for the dataset
+   */
+  public static synchronized void init(Configuration hadoopConf, String basePath) {
+    try {
+      if (instances.containsKey(basePath)) {
+        LOG.warn("Duplicate initialization of metadata for basePath " + basePath);
+      }
+      instances.put(basePath, new HoodieMetadataImpl(null, hadoopConf, basePath, null, true));
+    } catch (TableNotFoundException e) {
+      LOG.warn("Metadata table was not found at path " + getMetadataTableBasePath(basePath));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to initialize metadata table", e);
+    }
+  }
+
+  /**
+   * Update the metadata from a commit, delta-commit or compaction on the dataset.
+   *
+   * Updates are only performed if the Metadata Table was initialized in read-write mode.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} for the writer
+   * @param commitMetadata {@code HoodieCommitMetadata}
+   * @param instantTime The instant at which the commit took place
+   */
+  public static void update(HoodieWriteConfig writeConfig, HoodieCommitMetadata commitMetadata, String instantTime) {
+    if (writeConfig.useFileListingMetadata() && instances.containsKey(writeConfig.getBasePath())) {
+      instances.get(writeConfig.getBasePath()).update(commitMetadata, instantTime);
+    }
+  }
+
+  /**
+   * Update the metadata from a clean on the dataset.
+   *
+   * Updates are only performed if the Metadata Table was initialized in read-write mode.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} for the writer
+   * @param cleanerPlan {@code HoodieCleanerPlan}
+   * @param instantTime The instant at which the cleaner plan was generated
+   */
+  public static void update(HoodieWriteConfig writeConfig, HoodieCleanerPlan cleanerPlan, String instantTime) {
+    if (writeConfig.useFileListingMetadata() && instances.containsKey(writeConfig.getBasePath())) {
+      instances.get(writeConfig.getBasePath()).update(cleanerPlan, instantTime);
+    }
+  }
+
+  /**
+   * Update the metadata from a restore on the dataset.
+   *
+   * Updates are only performed if the Metadata Table was initialized in read-write mode.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} for the writer
+   * @param restoreMetadata {@code HoodieRestoreMetadata}
+   * @param instantTime The instant at which the restore took place
+   */
+  public static void update(HoodieWriteConfig writeConfig, HoodieRestoreMetadata restoreMetadata, String instantTime) {
+    if (writeConfig.useFileListingMetadata() && instances.containsKey(writeConfig.getBasePath())) {
+      instances.get(writeConfig.getBasePath()).update(restoreMetadata, instantTime);
+    }
+  }
+
+  /**
+   * Update the metadata from a rollback on the dataset.
+   *
+   * Updates are only performed if the Metadata Table was initialized in read-write mode.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} for the writer
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param instantTime The instant at which the rollback took place
+   */
+  public static void update(HoodieWriteConfig writeConfig, HoodieRollbackMetadata rollbackMetadata, String instantTime) {
+    if (writeConfig.useFileListingMetadata() && instances.containsKey(writeConfig.getBasePath())) {
+      instances.get(writeConfig.getBasePath()).update(rollbackMetadata, instantTime);
+    }
+  }
+
+  /**
+   * Return the base path of the Metadata Table.
+   *
+   * @param tableBasePath The base path of the dataset
+   */
+  public static String getMetadataTableBasePath(String tableBasePath) {
+    return HoodieMetadataImpl.getMetadataTableBasePath(tableBasePath);
+  }
+
+  /**
+   * Return the list of partitions in the dataset.
+   *
+   * If the Metadata Table is enabled, the listing is retrieved from the stored metadata. Otherwise, the list of
+   * partitions is retrieved directly from the underlying {@code FileSystem}.
+   *
+   * On any errors retrieving the listing from the metadata, defaults to using the file system listings.
+   *
+   * @param fs The {@code FileSystem}
+   * @param basePath Base path of the dataset
+   * @param assumeDatePartitioning True if the dataset uses date based partitioning
+   */
+  public static List<String> getAllPartitionPaths(FileSystem fs, String basePath, boolean assumeDatePartitioning)
+      throws IOException {
+    try {
+      if (instances.containsKey(basePath)) {
+        return instances.get(basePath).getAllPartitionPaths();
+      }
+    } catch (Exception e) {
+      LOG.error("Failed to retrive all partition paths from metadata: " + e);
+    }
+
+    return FSUtils.getAllPartitionPaths(fs, basePath, assumeDatePartitioning);

Review comment:
       Can we have a flag to control whether we error out with a HoodieIOException or fallback to listing? 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadata.java
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.exception.TableNotFoundException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+/**
+ * Defines the interface through which file listing metadata is created, updated and accessed. This metadata is
+ * saved within an internal Metadata Table.
+ *
+ * If file listing metadata is disabled, the functions default to using listStatus(...) RPC calls to retrieve
+ * file listings from the file system.
+ */
+public class HoodieMetadata {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadata.class);
+
+  // Instances of metadata for each basePath
+  private static Map<String, HoodieMetadataImpl> instances = new HashMap<>();
+
+  /**
+   * Initialize the Metadata Table.
+   *  - If the table does not exist it is created for the first time.
+   *  - If the table exists, it is synced with the instants on the dataset.
+   *
+   * @param jsc {@code JavaSparkContext}
+   * @param writeConfig {@code HoodieWriteConfig} for the writer from which relevant configuration is inherited
+   */
+  public static synchronized void init(JavaSparkContext jsc, HoodieWriteConfig writeConfig) {
+    try {
+      final String basePath = writeConfig.getBasePath();
+      if (instances.containsKey(basePath)) {
+        LOG.warn("Duplicate initialization of metadata for basePath " + basePath);
+      }
+      if (writeConfig.useFileListingMetadata()) {
+        instances.put(basePath, new HoodieMetadataImpl(jsc, jsc.hadoopConfiguration(), basePath, writeConfig, false));
+      } else {
+        if (!isMetadataTable(writeConfig.getBasePath())) {
+          LOG.info("Metadata table is disabled in write config.");
+        }
+      }
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to initialize metadata table", e);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table in read-only mode.
+   *
+   * If the table does not exist, a warning is logged and RPC calls are used to retrieve file listings
+   * from the file system. No updates are applied to the table and it is not synced.
+   *
+   * @param hadoopConf {@code Configuration}
+   * @param basePath The basePath for the dataset
+   */
+  public static synchronized void init(Configuration hadoopConf, String basePath) {
+    try {
+      if (instances.containsKey(basePath)) {
+        LOG.warn("Duplicate initialization of metadata for basePath " + basePath);
+      }
+      instances.put(basePath, new HoodieMetadataImpl(null, hadoopConf, basePath, null, true));
+    } catch (TableNotFoundException e) {
+      LOG.warn("Metadata table was not found at path " + getMetadataTableBasePath(basePath));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to initialize metadata table", e);
+    }
+  }
+
+  /**
+   * Update the metadata from a commit, delta-commit or compaction on the dataset.
+   *
+   * Updates are only performed if the Metadata Table was initialized in read-write mode.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} for the writer
+   * @param commitMetadata {@code HoodieCommitMetadata}
+   * @param instantTime The instant at which the commit took place
+   */
+  public static void update(HoodieWriteConfig writeConfig, HoodieCommitMetadata commitMetadata, String instantTime) {
+    if (writeConfig.useFileListingMetadata() && instances.containsKey(writeConfig.getBasePath())) {
+      instances.get(writeConfig.getBasePath()).update(commitMetadata, instantTime);
+    }
+  }
+
+  /**
+   * Update the metadata from a clean on the dataset.
+   *
+   * Updates are only performed if the Metadata Table was initialized in read-write mode.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} for the writer
+   * @param cleanerPlan {@code HoodieCleanerPlan}
+   * @param instantTime The instant at which the cleaner plan was generated
+   */
+  public static void update(HoodieWriteConfig writeConfig, HoodieCleanerPlan cleanerPlan, String instantTime) {
+    if (writeConfig.useFileListingMetadata() && instances.containsKey(writeConfig.getBasePath())) {
+      instances.get(writeConfig.getBasePath()).update(cleanerPlan, instantTime);
+    }
+  }
+
+  /**
+   * Update the metadata from a restore on the dataset.
+   *
+   * Updates are only performed if the Metadata Table was initialized in read-write mode.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} for the writer
+   * @param restoreMetadata {@code HoodieRestoreMetadata}
+   * @param instantTime The instant at which the restore took place
+   */
+  public static void update(HoodieWriteConfig writeConfig, HoodieRestoreMetadata restoreMetadata, String instantTime) {
+    if (writeConfig.useFileListingMetadata() && instances.containsKey(writeConfig.getBasePath())) {
+      instances.get(writeConfig.getBasePath()).update(restoreMetadata, instantTime);
+    }
+  }
+
+  /**
+   * Update the metadata from a rollback on the dataset.
+   *
+   * Updates are only performed if the Metadata Table was initialized in read-write mode.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} for the writer
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param instantTime The instant at which the rollback took place
+   */
+  public static void update(HoodieWriteConfig writeConfig, HoodieRollbackMetadata rollbackMetadata, String instantTime) {
+    if (writeConfig.useFileListingMetadata() && instances.containsKey(writeConfig.getBasePath())) {
+      instances.get(writeConfig.getBasePath()).update(rollbackMetadata, instantTime);
+    }
+  }
+
+  /**
+   * Return the base path of the Metadata Table.
+   *
+   * @param tableBasePath The base path of the dataset
+   */
+  public static String getMetadataTableBasePath(String tableBasePath) {
+    return HoodieMetadataImpl.getMetadataTableBasePath(tableBasePath);
+  }
+
+  /**
+   * Return the list of partitions in the dataset.
+   *
+   * If the Metadata Table is enabled, the listing is retrieved from the stored metadata. Otherwise, the list of
+   * partitions is retrieved directly from the underlying {@code FileSystem}.
+   *
+   * On any errors retrieving the listing from the metadata, defaults to using the file system listings.
+   *
+   * @param fs The {@code FileSystem}
+   * @param basePath Base path of the dataset
+   * @param assumeDatePartitioning True if the dataset uses date based partitioning
+   */
+  public static List<String> getAllPartitionPaths(FileSystem fs, String basePath, boolean assumeDatePartitioning)
+      throws IOException {
+    try {
+      if (instances.containsKey(basePath)) {
+        return instances.get(basePath).getAllPartitionPaths();
+      }
+    } catch (Exception e) {
+      LOG.error("Failed to retrive all partition paths from metadata: " + e);

Review comment:
       LOG.error(msg, e) everywhere please. we will get the stack trace that way. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)

Review comment:
       this has to eventually change when writing bloom filters parallely via Spark for eg

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";

Review comment:
       can we rename this to `filesystem_metadata`  or `fs_metadata`. It will be easy to then name subsequent partitions as `filestats_metadata`, `bloomfilter_metadata` etc 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";

Review comment:
       we use something similar for bootstrap. So wondering if this will be an issue for a bootstrapped table with metadata. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");

Review comment:
       this is okay for now. but can we file a follow on for async compaction of metadata as well. Do you see any fundamental issues with using async compaction on metadata table

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied

Review comment:
       async compaction on the data timeline can have an instant time that is lower than other following actions like delta commits, cleans that happen while compaction is pending. Do we handle that scenario? i.e we don't miss adding the new base files produced by the compaction. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataIndex.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIndexException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.index.HoodieIndexUtils;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import scala.Tuple2;
+
+/**
+ * An Index which is specific to the Hoodie Metadata Table.
+ *
+ * Hoodie Metadata Table saves information in specific named partitions. So the partition and the base file location
+ * during indexing can be determined by looking at the record itself. This speeds up the indexing during updates.
+ */
+public class HoodieMetadataIndex<T extends HoodieMetadataPayload> extends HoodieIndex<T> {

Review comment:
       We can also the prepped* APIs and avoid the index completely?

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)

Review comment:
       see my comment from before. I think we can just support the same write configs for metadata table , by letting the user pass in `hoodie.metadata.<existing_config>`, this way we can just constcut he writeConfig object for metadata easily from then parent properties itself, without having to decide a default for each parameter at this level.  

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {

Review comment:
       anyway to reuse this code?

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";

Review comment:
       any way to reuse these without having to redefine again

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;

Review comment:
       I think it will make this file shorter/more abstracted. if we pulled the objects needed for reading writing to a separate class like `FileSystemMetadata`, leaving this class to operate more at the table level.  

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;

Review comment:
       Seems like you will use this timestamp for the intial base file. This is same as HoodieTimeline.INIT_INSTANT_TS. Can we reuse that constant? and more importantly, would this cause any issues when we use the bootstrap feature? 
   In this scenario, there will be a 0000000000 commit on the dataset timeline. and you will attempt to also create the same instant first during initing and then subsequently for the actual write? 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)

Review comment:
       please remove if dead code.

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");

Review comment:
       if we fail doing this i.e the initial commit with all the existing files. the metadata table will be created, but no commits. how do we detect this and attempt reiniting next time around?

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataIndex.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIndexException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.index.HoodieIndexUtils;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import scala.Tuple2;
+
+/**
+ * An Index which is specific to the Hoodie Metadata Table.
+ *
+ * Hoodie Metadata Table saves information in specific named partitions. So the partition and the base file location
+ * during indexing can be determined by looking at the record itself. This speeds up the indexing during updates.
+ */
+public class HoodieMetadataIndex<T extends HoodieMetadataPayload> extends HoodieIndex<T> {
+
+  public HoodieMetadataIndex(HoodieWriteConfig config) {
+    super(config);
+  }
+
+  /**
+   * Returns an RDD mapping each HoodieKey with a partitionPath/fileID which contains it. Option.Empty if the key is not
+   * found.
+   *
+   * @param hoodieKeys  keys to lookup
+   * @param jsc         spark context
+   * @param hoodieTable hoodie table object
+   */
+  @Override
+  public JavaPairRDD<HoodieKey, Option<Pair<String, String>>> fetchRecordLocation(JavaRDD<HoodieKey> hoodieKeys,
+                                                                                  JavaSparkContext jsc,
+                                                                                  HoodieTable<T> hoodieTable) {
+    JavaPairRDD<String, String> partitionRecordKeyPairRDD =
+        hoodieKeys.mapToPair(key -> new Tuple2<>(key.getPartitionPath(), key.getRecordKey()));
+
+    // Lookup location of all the hoodie keys
+    JavaPairRDD<HoodieKey, HoodieRecordLocation> recordKeyLocationRDD = lookupIndex(hoodieKeys, jsc, hoodieTable);
+    JavaPairRDD<HoodieKey, String> keyHoodieKeyPairRDD = hoodieKeys.mapToPair(key -> new Tuple2<>(key, null));
+
+    return keyHoodieKeyPairRDD.leftOuterJoin(recordKeyLocationRDD).mapToPair(keyLoc -> {
+      Option<Pair<String, String>> partitionPathFileidPair;
+      if (keyLoc._2._2.isPresent()) {
+        partitionPathFileidPair = Option.of(Pair.of(keyLoc._1().getPartitionPath(), keyLoc._2._2.get().getFileId()));
+      } else {
+        partitionPathFileidPair = Option.empty();
+      }
+      return new Tuple2<>(keyLoc._1, partitionPathFileidPair);
+    });
+  }
+
+  /**
+   * Tag each record with its current location (partition / fileId).
+   *
+   * @param recordRDD   records to tag
+   * @param jsc         spark context
+   * @param hoodieTable hoodie table object
+   */
+  @Override
+  public JavaRDD<HoodieRecord<T>> tagLocation(JavaRDD<HoodieRecord<T>> recordRDD, JavaSparkContext jsc,
+                                              HoodieTable<T> hoodieTable) throws HoodieIndexException {
+    JavaPairRDD<HoodieKey, HoodieRecord> keyRecordPairRDD = recordRDD.mapToPair(r -> new Tuple2<>(r.getKey(), r));
+    JavaPairRDD<HoodieKey, HoodieRecordLocation> keyLocationPairRDD =
+        lookupIndex(keyRecordPairRDD.keys(), jsc, hoodieTable);
+
+    return keyRecordPairRDD.leftOuterJoin(keyLocationPairRDD).values().map(v1 -> {
+      return HoodieIndexUtils.getTaggedRecord(v1._1, Option.ofNullable(v1._2.orNull()));
+    });
+  }
+
+  /**
+   * Update the index (if required) from the results of the writes into the hoodie table.
+   *
+   * @param writeStatusRDD  status of the writes
+   * @param jsc             spark context
+   * @param hoodieTable     hoodie table object
+   */
+  @Override
+  public JavaRDD<WriteStatus> updateLocation(JavaRDD<WriteStatus> writeStatusRDD, JavaSparkContext jsc,
+                                             HoodieTable<T> hoodieTable)
+      throws HoodieIndexException {
+    // No update required for metadata index
+    return writeStatusRDD;
+  }
+
+  /**
+   * Lookup the location for each hoodie key and return the pair<key, location> for all record keys already
+   * present and drop the record keys if not present.
+   */
+  private JavaPairRDD<HoodieKey, HoodieRecordLocation> lookupIndex(JavaRDD<HoodieKey> hoodieKeys,
+                                                                   JavaSparkContext jsc,
+                                                                   HoodieTable<T> hoodieTable) {
+    SliceView fsView = hoodieTable.getSliceView();
+    List<HoodieBaseFile> baseFiles = fsView.getLatestFileSlices(HoodieMetadataImpl.METADATA_PARTITION_NAME)

Review comment:
       this has to be derived from the input keys eventually, to support updates to different metadata partitions?

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};

Review comment:
       can we move this to its own method? to help keep this method shorter

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)
+            && (metaSyncTimestamp.isEmpty() || HoodieTimeline.compareTimestamps(i.getTimestamp(),
+                HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp)))
+        .collect(Collectors.toMap(HoodieInstant::getTimestamp, Function.identity()));
+
+    List<HoodieInstant> instantsToSync = new LinkedList<>();
+    datasetInstants.forEach(instant -> {
+      if (metadataInstantMap.containsKey(instant.getTimestamp())) {
+        // instant already synced to metadata table
+        if (!instantsToSync.isEmpty()) {
+          LOG.warn("Found out-of-order already synced instant " + instant + ". Instants to sync=" + instantsToSync);
+        }
+      } else {
+        instantsToSync.add(instant);
+      }
+    });
+
+    // TODO: async clean and async compaction are not handled here. They have a timestamp which is in the past
+    // (when the operation was scheduled) and even on completion they retain their old timestamp.
+    return instantsToSync;
+  }
+
+  /**
+   * Update from {@code HoodieCommitMetadata}.
+   *
+   * @param commitMetadata {@code HoodieCommitMetadata}
+   * @param instantTime Timestamp at which the commit was performed
+   */
+  void update(HoodieCommitMetadata commitMetadata, String instantTime) {

Review comment:
       I feel these update methods can be moved to a new class, which encapsulates all code for reading/writing to file system metadata. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * This is a payload which saves information about a single entry in the Metadata Table. The type of the entry is
+ * determined by the "type" saved within the record. The following types of entries are saved:
+ *
+ *   1. List of partitions: There is a single such record
+ *         key="__all_partitions__"
+ *         filenameToSizeMap={"2020/01/01": 0, "2020/01/02": 0, ...}
+ *
+ *   2. List of files in a Partition: There is one such record for each partition
+ *         key=Partition name
+ *         filenameToSizeMap={"file1.parquet": 12345, "file2.parquet": 56789, "file1.log": 9876,
+ *                            "file0.parquet": -1, ...}
+ *
+ *      For deleted files, -1 is used as the size.
+ *
+ *  During compaction on the table, the deletions are merged with additions and hence pruned.
+ */
+public class HoodieMetadataPayload implements HoodieRecordPayload<HoodieMetadataPayload> {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataPayload.class);
+
+  // Represents the size stored for a deleted file
+  private static final long DELETED_FILE_SIZE = -1;
+
+  // Key and type for the metadata record
+  private final String metadataKey;
+  private final PayloadType type;
+
+  // Filenames which are part of this record
+  // key=filename, value=file size (or DELETED_FILE_SIZE to represent a deleted file)
+  private final Map<String, Long> filenameMap = new HashMap<>();
+
+  // Type of the metadata record
+  public enum PayloadType {
+    PARTITION_LIST(1),        // list of partitions
+    PARTITION_FILES(2);       // list of files in a partition
+
+    private final int value;

Review comment:
       we can just use the enum ordinal right? without introducing an explicit `value` field. are you concerned about someone reordering? 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table

Review comment:
       +1 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))

Review comment:
       but when we are syncing from timeline. its better to not rely on inflight instants right? for e.g what if there is a .inflight clean that never does completed and the user redeployed with a different config for cleaning, which results in a different clean plan the next time. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");

Review comment:
       `    boolean exists = fs.exists(new Path(metadataBasePath));` , I am saying exists will be equal to true, even if we fail to do the initing and fail midway?

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)
+            && (metaSyncTimestamp.isEmpty() || HoodieTimeline.compareTimestamps(i.getTimestamp(),
+                HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp)))
+        .collect(Collectors.toMap(HoodieInstant::getTimestamp, Function.identity()));
+
+    List<HoodieInstant> instantsToSync = new LinkedList<>();
+    datasetInstants.forEach(instant -> {
+      if (metadataInstantMap.containsKey(instant.getTimestamp())) {
+        // instant already synced to metadata table
+        if (!instantsToSync.isEmpty()) {
+          LOG.warn("Found out-of-order already synced instant " + instant + ". Instants to sync=" + instantsToSync);
+        }
+      } else {
+        instantsToSync.add(instant);
+      }
+    });
+
+    // TODO: async clean and async compaction are not handled here. They have a timestamp which is in the past
+    // (when the operation was scheduled) and even on completion they retain their old timestamp.
+    return instantsToSync;
+  }
+
+  /**
+   * Update from {@code HoodieCommitMetadata}.
+   *
+   * @param commitMetadata {@code HoodieCommitMetadata}
+   * @param instantTime Timestamp at which the commit was performed
+   */
+  void update(HoodieCommitMetadata commitMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    commitMetadata.getPartitionToWriteStats().forEach((partition, writeStats) -> {
+      Map<String, Long> newFiles = new HashMap<>(writeStats.size());
+      writeStats.forEach(hoodieWriteStat -> {
+        String pathWithPartition = hoodieWriteStat.getPath();
+        if (pathWithPartition == null) {
+          // Empty partition
+          return;
+        }
+        String filename = pathWithPartition.substring(partition.length() + 1);
+        ValidationUtils.checkState(!newFiles.containsKey(filename), "Duplicate files in HoodieCommitMetadata");
+        newFiles.put(filename, hoodieWriteStat.getTotalWriteBytes());
+      });
+
+      // New files added to a partition
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.of(newFiles),
+          Option.empty());
+      records.add(record);
+    });
+
+    // New partitions created
+    HoodieRecord record = HoodieMetadataPayload.createPartitionListRecord(
+        new ArrayList<String>(commitMetadata.getPartitionToWriteStats().keySet()));
+    records.add(record);
+
+    LOG.info("Updating at " + instantTime + " from Commit/" + commitMetadata.getOperationType()
+        + ". #partitions_updated=" + records.size());
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanerPlan}.
+   *
+   * @param cleanerPlan {@code HoodieCleanerPlan}
+   * @param instantTime Timestamp at which the clean plan was generated
+   */
+  void update(HoodieCleanerPlan cleanerPlan, String instantTime) {
+    // TODO: Cleaner update is called even before the operation has completed. Hence, this function may be called
+    // multiple times and we dont need to update metadata again.
+    HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+    long cnt = timeline.filterCompletedInstants().getInstants().filter(i -> i.getTimestamp().equals(instantTime)).count();
+    if (cnt == 1) {
+      LOG.info("Ignoring update from cleaner plan for already completed instant " + instantTime);
+      return;
+    }
+
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+    cleanerPlan.getFilePathsToBeDeletedPerPartition().forEach((partition, deletedPathInfo) -> {
+      fileDeleteCount[0] += deletedPathInfo.size();
+
+      // Files deleted from a partition
+      List<String> deletedFilenames = deletedPathInfo.stream().map(p -> new Path(p.getFilePath()).getName())
+          .collect(Collectors.toList());
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(deletedFilenames));
+      records.add(record);
+    });
+
+    LOG.info("Updating at " + instantTime + " from CleanerPlan. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanMetadata}.
+   *
+   * @param cleanMetadata {@code HoodieCleanMetadata}
+   * @param instantTime Timestamp at which the clean was completed
+   */
+  private void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+
+    cleanMetadata.getPartitionMetadata().forEach((partition, partitionMetadata) -> {
+      // Files deleted from a partition
+      List<String> deletedFiles = partitionMetadata.getSuccessDeleteFiles();
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(new ArrayList<>(deletedFiles)));
+
+      records.add(record);
+      fileDeleteCount[0] += deletedFiles.size();
+    });
+
+    LOG.info("Updating at " + instantTime + " from Clean. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieRestoreMetadata}.
+   *
+   * @param restoreMetadata {@code HoodieRestoreMetadata}
+   * @param instantTime Timestamp at which the restore was performed
+   */
+  void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    restoreMetadata.getHoodieRestoreMetadata().values().forEach(rms -> {
+      rms.forEach(rm -> processRollbackMetadata(rm, partitionToDeletedFiles));
+    });
+
+    commitRollbackDeletes(partitionToDeletedFiles, instantTime, "Restore");
+  }
+
+  /**
+   * Update from {@code HoodieRollbackMetadata}.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param instantTime Timestamp at which the rollback was performed
+   */
+  void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    processRollbackMetadata(rollbackMetadata, partitionToDeletedFiles);
+    commitRollbackDeletes(partitionToDeletedFiles, instantTime, "Rollback");
+  }
+
+  /**
+   * Retrives the partition and deleted files from {@code HoodieRollbackMetadata}.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param partitionToDeletedFiles The {@code Map} to fill with partition and deleted files.
+   */
+  private void processRollbackMetadata(HoodieRollbackMetadata rollbackMetadata,
+                                       Map<String, List<String>> partitionToDeletedFiles) {
+    rollbackMetadata.getPartitionMetadata().values().forEach(pm -> {
+      // Extract deleted file name from the absolute paths saved in getSuccessDeleteFiles()
+      List<String> deletedFiles = pm.getSuccessDeleteFiles().stream().map(p -> new Path(p).getName())
+          .collect(Collectors.toList());
+      if (partitionToDeletedFiles.containsKey(pm.getPartitionPath())) {
+        partitionToDeletedFiles.get(pm.getPartitionPath()).addAll(deletedFiles);
+      } else {
+        partitionToDeletedFiles.put(pm.getPartitionPath(), new ArrayList<>(deletedFiles));
+      }
+    });
+  }
+
+  /**
+   * Create file delete records and commit.
+   *
+   * @param partitionToDeletedFiles {@code Map} of partitions and the deleted files
+   * @param instantTime Timestamp at which the deletes took place
+   * @param operation Type of the operation which caused the files to be deleted
+   */
+  private void commitRollbackDeletes(Map<String, List<String>> partitionToDeletedFiles, String instantTime,
+                                     String operation) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+    partitionToDeletedFiles.forEach((partition, deletedFiles) -> {
+      // Rollbacks deletes instants from timeline. The instant being rolled-back may not have been synced to the
+      // metadata table. Hence, the deleted filed need to be checked against the metadata.
+      try {
+        int origCount = deletedFiles.size();
+        FileStatus[] existingStatuses = getAllFilesInPartition(new Path(basePath, partition));
+        Set<String> currentFiles =
+            Arrays.stream(existingStatuses).map(s -> s.getPath().getName()).collect(Collectors.toSet());
+        deletedFiles.removeIf(f -> !currentFiles.contains(f));
+        if (deletedFiles.size() != origCount) {
+          LOG.warn("Some Files to be deleted as part of " + operation + " at " + instantTime + " were not found in the "
+              + " metadata for partition " + partition
+              + ". To delete = " + origCount + ", found=" + deletedFiles.size());
+        }
+
+        // Commit this record even if there are no deleted files. This will create a delta-commit corresponding to
+        // the instant being synced.
+        fileDeleteCount[0] += deletedFiles.size();
+
+        // Files deleted from a partition
+        HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+            Option.of(new ArrayList<>(deletedFiles)));
+        records.add(record);
+
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to commit rollback deletes at instant " + instantTime, e);
+      }
+    });
+
+    if (!records.isEmpty()) {
+      LOG.info("Updating at " + instantTime + " from " + operation + ". #partitions_updated=" + records.size()
+          + ", #files_deleted=" + fileDeleteCount);
+      commit(records, instantTime, true);
+    }
+  }
+
+  /**
+   * Commit the {@code HoodieRecord}s to Metadata Table as a new delta-commit.
+   *
+   * @param records The records to commit
+   * @param instantTime The timestamp of instant to create
+   * @param performUpdate If True, records are upserted. If False, records are inserted
+   */
+  private synchronized void commit(List<HoodieRecord> records, String instantTime, boolean performUpdate) {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be committed in readonly mode");
+    ValidationUtils.checkArgument(!records.isEmpty());
+
+    // Close all readers
+    if (basefileReader != null) {
+      basefileReader.close();
+      basefileReader = null;
+    }
+    logRecordScanner = null;
+
+    try (HoodieWriteClient writeClient = new HoodieWriteClient(jsc, config, true)) {
+      writeClient.startCommitWithTime(instantTime);
+      List<WriteStatus> statuses;
+      if (performUpdate) {
+        statuses = writeClient.upsert(jsc.parallelize(records), instantTime).collect();
+      } else {
+        statuses = writeClient.insert(jsc.parallelize(records), instantTime).collect();
+      }
+      statuses.forEach(writeStatus -> {
+        if (writeStatus.hasErrors()) {
+          throw new HoodieMetadataException("Failed to commit metadata table records at instant " + instantTime);
+        }
+      });
+    }
+  }
+
+  static String getMetadataTableBasePath(String tableBasePath) {
+    return tableBasePath + Path.SEPARATOR + METADATA_TABLE_REL_PATH;
+  }
+
+  /**
+   * Returns a list of all partitions.
+   */
+  List<String> getAllPartitionPaths() throws IOException {
+    final Timer.Context context = this.metrics.getMetadataCtx(LOOKUP_PARTITIONS_STR);
+
+    Option<HoodieRecord<HoodieMetadataPayload>> hoodieRecord = getMergedRecordByKey(HoodieMetadataImpl.RECORDKEY_PARTITION_LIST);
+    List<String> partitions = Collections.emptyList();
+    if (hoodieRecord.isPresent()) {
+      if (!hoodieRecord.get().getData().getDeletions().isEmpty()) {
+        throw new HoodieMetadataException("Metadata partition list record is inconsistent: "
+            + hoodieRecord.get().getData());
+      }
+
+      partitions = hoodieRecord.get().getData().getFilenames();
+    }
+
+    ++numLookupsAllPartitions;
+
+    long validateTime = 0;
+    if (validateLookups) {
+      // Validate the Metadata Table data by listing the partitions from the file system
+      final Timer.Context contextValidate = this.metrics.getMetadataCtx(VALIDATE_PARTITIONS_STR);
+      List<String> actualPartitions  = FSUtils.getAllPartitionPaths(metaClient.getFs(), basePath, false);
+
+      Collections.sort(actualPartitions);
+      Collections.sort(partitions);
+      if (!actualPartitions.equals(partitions)) {
+        LOG.error("Validation of metadata partition list failed. Lists do not match.");
+        LOG.error("Partitions from metadata: " + Arrays.toString(partitions.toArray()));
+        LOG.error("Partitions from file system: " + Arrays.toString(actualPartitions.toArray()));
+
+        ++numValidationErrors;
+        metrics.updateMetadataMetrics(VALIDATE_ERRORS_STR, 0, numValidationErrors);
+      }
+
+      // Return the direct listing as it should be correct
+      partitions = actualPartitions;
+
+      if (contextValidate != null) {
+        validateTime = metrics.getDurationInMs(contextValidate.stop());
+        metrics.updateMetadataMetrics(VALIDATE_PARTITIONS_STR, validateTime, numLookupsAllPartitions);
+      }
+    }
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop()) - validateTime;
+      metrics.updateMetadataMetrics(LOOKUP_PARTITIONS_STR, durationInMs, numLookupsAllPartitions);
+    }
+
+    return partitions;
+  }
+
+  /**
+   * Return all the files from the partition.
+   *
+   * @param partitionPath The absolute path of the partition
+   */
+  FileStatus[] getAllFilesInPartition(Path partitionPath) throws IOException {
+    final Timer.Context context = this.metrics.getMetadataCtx(LOOKUP_FILES_STR);
+
+    String partitionName = FSUtils.getRelativePartitionPath(new Path(basePath), partitionPath);
+
+    Option<HoodieRecord<HoodieMetadataPayload>> hoodieRecord = getMergedRecordByKey(partitionName);
+    FileStatus[] statuses = {};
+    if (hoodieRecord.isPresent()) {
+      if (!hoodieRecord.get().getData().getDeletions().isEmpty()) {
+        throw new HoodieMetadataException("Metadata record for partition " + partitionName + " is inconsistent: "
+              + hoodieRecord.get().getData());
+      }
+      statuses = hoodieRecord.get().getData().getFileStatuses(partitionPath);
+    }
+
+    ++numLookupsFilesInPartition;
+
+    long validateTime = 0;
+    if (validateLookups) {
+      // Validate the Metadata Table data by listing the partitions from the file system
+      final Timer.Context contextValidate = this.metrics.getMetadataCtx(VALIDATE_FILES_STR);
+      validateTime = System.currentTimeMillis();
+      // Compare the statuses from metadata to those from listing directly from the file system.
+      // Ignore partition metadata file
+      FileStatus[] directStatuses = metaClient.getFs().listStatus(partitionPath,
+          p -> !p.getName().equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE));
+      List<String> directFilenames = Arrays.stream(directStatuses)
+          .map(s -> s.getPath().getName()).collect(Collectors.toList());
+      List<String> metadataFilenames = Arrays.stream(statuses)
+          .map(s -> s.getPath().getName()).collect(Collectors.toList());
+
+      Collections.sort(metadataFilenames);
+      Collections.sort(directFilenames);
+      if (!metadataFilenames.equals(directFilenames)) {
+        LOG.error("Validation of metadata file listing for partition " + partitionName + " failed.");
+        LOG.error("File list from metadata: " + Arrays.toString(metadataFilenames.toArray()));
+        LOG.error("File list from direct listing: " + Arrays.toString(directFilenames.toArray()));
+
+        ++numValidationErrors;
+        metrics.updateMetadataMetrics(VALIDATE_ERRORS_STR, 0, numValidationErrors);
+      }
+
+      // Return the direct listing as it should be correct
+      statuses = directStatuses;
+
+      if (contextValidate != null) {
+        validateTime = metrics.getDurationInMs(contextValidate.stop());
+        metrics.updateMetadataMetrics(VALIDATE_FILES_STR, validateTime, numLookupsFilesInPartition);
+      }
+    }
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop()) - validateTime;
+      metrics.updateMetadataMetrics(LOOKUP_FILES_STR, durationInMs, numLookupsFilesInPartition);
+    }
+
+    return statuses;
+  }
+
+  /**
+   * Open readers to the base and log files.
+   */
+  private synchronized void openBaseAndLogFiles() throws IOException {
+    if (logRecordScanner != null) {
+      // Already opened
+      return;
+    }
+
+    final Timer.Context context = this.metrics.getMetadataCtx(SCAN_STR);
+
+    // Find the latest file slice
+    HoodieTimeline timeline = metaClient.reloadActiveTimeline();
+    HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+    List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+    ValidationUtils.checkArgument(latestSlices.size() == 1);
+
+    // If the base file is present then create a reader
+    Option<HoodieBaseFile> basefile = latestSlices.get(0).getBaseFile();
+    if (basefile.isPresent()) {
+      String basefilePath = basefile.get().getPath();
+      basefileReader = HoodieFileReaderFactory.getFileReader(hadoopConf, new Path(basefilePath));
+
+      LOG.info("Opened metadata base file from " + basefilePath + " at instant " + basefile.get().getCommitTime());
+    }
+
+    // Open the log record scanner using the log files from the latest file slice
+    List<String> logFilePaths = latestSlices.get(0).getLogFiles().map(o -> o.getPath().toString())
+        .collect(Collectors.toList());
+    Option<HoodieInstant> lastInstant = timeline.filterCompletedInstants().lastInstant();
+    String latestTimestamp = lastInstant.isPresent() ? lastInstant.get().getTimestamp() : "";
+
+    logRecordScanner =
+        new HoodieMetadataMergedLogRecordScanner(FSUtils.getFs(basePath, hadoopConf), metadataBasePath,
+            logFilePaths, schema, latestTimestamp, maxMemorySizeInBytes, bufferSize,
+            config.getSpillableMapBasePath(), null);
+
+    LOG.info("Opened metadata log files from " + logFilePaths + " at instant " + lastInstant);
+
+    ++numScans;
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      metrics.updateMetadataMetrics(SCAN_STR, durationInMs, numScans);
+    }
+  }
+
+  /**
+   * Retrieve the merged {@code HoodieRecord} mapped to the given key.
+   *
+   * @param key The key of the record
+   */
+  private Option<HoodieRecord<HoodieMetadataPayload>> getMergedRecordByKey(String key) throws IOException {
+    openBaseAndLogFiles();
+
+    // Retrieve record from base file
+    HoodieRecord<HoodieMetadataPayload> hoodieRecord = null;
+    if (basefileReader != null) {
+      Option<GenericRecord> baseRecord = basefileReader.getRecordByKey(key);
+      if (baseRecord.isPresent()) {
+        hoodieRecord = SpillableMapUtils.convertToHoodieRecordPayload((GenericRecord) baseRecord.get(),
+            metaClient.getTableConfig().getPayloadClass());
+      }
+    }
+
+    // Retrieve record from log file
+    Option<HoodieRecord<HoodieMetadataPayload>> logHoodieRecord = logRecordScanner.getRecordByKey(key);
+    if (logHoodieRecord.isPresent()) {
+      if (hoodieRecord != null) {
+        // Merge the payloads
+        HoodieRecordPayload mergedPayload = hoodieRecord.getData().preCombine(logHoodieRecord.get().getData());
+        hoodieRecord = new HoodieRecord(hoodieRecord.getKey(), mergedPayload);
+      } else {
+        hoodieRecord = logHoodieRecord.get();
+      }
+    }
+
+    return Option.ofNullable(hoodieRecord);
+  }
+
+  /**
+   * Return {@code True} if all Instants from the dataset have been synced with the Metadata Table.
+   */
+  boolean isInSync() {
+    // There should not be any instants to sync
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    return instantsToSync.isEmpty();
+  }
+
+  /**
+   * Return the timestamp of the latest compaction instant.
+   */
+  Option<String> getLatestCompactionTimestamp() {
+    HoodieTimeline timeline = metaClient.reloadActiveTimeline();
+    Option<HoodieInstant> lastCompactionInstant = timeline.filterCompletedInstants()
+        .filter(i -> i.getAction().equals(HoodieTimeline.COMMIT_ACTION)).lastInstant();
+
+    if (lastCompactionInstant.isPresent()) {

Review comment:
       Option.map etc. will make this a single line. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)
+            && (metaSyncTimestamp.isEmpty() || HoodieTimeline.compareTimestamps(i.getTimestamp(),
+                HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp)))
+        .collect(Collectors.toMap(HoodieInstant::getTimestamp, Function.identity()));
+
+    List<HoodieInstant> instantsToSync = new LinkedList<>();
+    datasetInstants.forEach(instant -> {
+      if (metadataInstantMap.containsKey(instant.getTimestamp())) {
+        // instant already synced to metadata table
+        if (!instantsToSync.isEmpty()) {
+          LOG.warn("Found out-of-order already synced instant " + instant + ". Instants to sync=" + instantsToSync);
+        }
+      } else {
+        instantsToSync.add(instant);
+      }
+    });
+
+    // TODO: async clean and async compaction are not handled here. They have a timestamp which is in the past
+    // (when the operation was scheduled) and even on completion they retain their old timestamp.
+    return instantsToSync;
+  }
+
+  /**
+   * Update from {@code HoodieCommitMetadata}.
+   *
+   * @param commitMetadata {@code HoodieCommitMetadata}
+   * @param instantTime Timestamp at which the commit was performed
+   */
+  void update(HoodieCommitMetadata commitMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    commitMetadata.getPartitionToWriteStats().forEach((partition, writeStats) -> {
+      Map<String, Long> newFiles = new HashMap<>(writeStats.size());
+      writeStats.forEach(hoodieWriteStat -> {
+        String pathWithPartition = hoodieWriteStat.getPath();
+        if (pathWithPartition == null) {
+          // Empty partition
+          return;
+        }
+        String filename = pathWithPartition.substring(partition.length() + 1);
+        ValidationUtils.checkState(!newFiles.containsKey(filename), "Duplicate files in HoodieCommitMetadata");
+        newFiles.put(filename, hoodieWriteStat.getTotalWriteBytes());
+      });
+
+      // New files added to a partition
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.of(newFiles),
+          Option.empty());
+      records.add(record);
+    });
+
+    // New partitions created
+    HoodieRecord record = HoodieMetadataPayload.createPartitionListRecord(
+        new ArrayList<String>(commitMetadata.getPartitionToWriteStats().keySet()));
+    records.add(record);
+
+    LOG.info("Updating at " + instantTime + " from Commit/" + commitMetadata.getOperationType()
+        + ". #partitions_updated=" + records.size());
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanerPlan}.
+   *
+   * @param cleanerPlan {@code HoodieCleanerPlan}
+   * @param instantTime Timestamp at which the clean plan was generated
+   */
+  void update(HoodieCleanerPlan cleanerPlan, String instantTime) {
+    // TODO: Cleaner update is called even before the operation has completed. Hence, this function may be called
+    // multiple times and we dont need to update metadata again.
+    HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+    long cnt = timeline.filterCompletedInstants().getInstants().filter(i -> i.getTimestamp().equals(instantTime)).count();
+    if (cnt == 1) {
+      LOG.info("Ignoring update from cleaner plan for already completed instant " + instantTime);
+      return;
+    }
+
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+    cleanerPlan.getFilePathsToBeDeletedPerPartition().forEach((partition, deletedPathInfo) -> {
+      fileDeleteCount[0] += deletedPathInfo.size();
+
+      // Files deleted from a partition
+      List<String> deletedFilenames = deletedPathInfo.stream().map(p -> new Path(p.getFilePath()).getName())
+          .collect(Collectors.toList());
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(deletedFilenames));
+      records.add(record);
+    });
+
+    LOG.info("Updating at " + instantTime + " from CleanerPlan. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanMetadata}.
+   *
+   * @param cleanMetadata {@code HoodieCleanMetadata}
+   * @param instantTime Timestamp at which the clean was completed
+   */
+  private void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+
+    cleanMetadata.getPartitionMetadata().forEach((partition, partitionMetadata) -> {
+      // Files deleted from a partition
+      List<String> deletedFiles = partitionMetadata.getSuccessDeleteFiles();
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(new ArrayList<>(deletedFiles)));
+
+      records.add(record);
+      fileDeleteCount[0] += deletedFiles.size();
+    });
+
+    LOG.info("Updating at " + instantTime + " from Clean. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieRestoreMetadata}.
+   *
+   * @param restoreMetadata {@code HoodieRestoreMetadata}
+   * @param instantTime Timestamp at which the restore was performed
+   */
+  void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    restoreMetadata.getHoodieRestoreMetadata().values().forEach(rms -> {
+      rms.forEach(rm -> processRollbackMetadata(rm, partitionToDeletedFiles));
+    });
+
+    commitRollbackDeletes(partitionToDeletedFiles, instantTime, "Restore");
+  }
+
+  /**
+   * Update from {@code HoodieRollbackMetadata}.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param instantTime Timestamp at which the rollback was performed
+   */
+  void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    processRollbackMetadata(rollbackMetadata, partitionToDeletedFiles);
+    commitRollbackDeletes(partitionToDeletedFiles, instantTime, "Rollback");
+  }
+
+  /**
+   * Retrives the partition and deleted files from {@code HoodieRollbackMetadata}.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param partitionToDeletedFiles The {@code Map} to fill with partition and deleted files.
+   */
+  private void processRollbackMetadata(HoodieRollbackMetadata rollbackMetadata,
+                                       Map<String, List<String>> partitionToDeletedFiles) {
+    rollbackMetadata.getPartitionMetadata().values().forEach(pm -> {
+      // Extract deleted file name from the absolute paths saved in getSuccessDeleteFiles()
+      List<String> deletedFiles = pm.getSuccessDeleteFiles().stream().map(p -> new Path(p).getName())
+          .collect(Collectors.toList());
+      if (partitionToDeletedFiles.containsKey(pm.getPartitionPath())) {
+        partitionToDeletedFiles.get(pm.getPartitionPath()).addAll(deletedFiles);
+      } else {
+        partitionToDeletedFiles.put(pm.getPartitionPath(), new ArrayList<>(deletedFiles));
+      }
+    });
+  }
+
+  /**
+   * Create file delete records and commit.
+   *
+   * @param partitionToDeletedFiles {@code Map} of partitions and the deleted files
+   * @param instantTime Timestamp at which the deletes took place
+   * @param operation Type of the operation which caused the files to be deleted
+   */
+  private void commitRollbackDeletes(Map<String, List<String>> partitionToDeletedFiles, String instantTime,
+                                     String operation) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+    partitionToDeletedFiles.forEach((partition, deletedFiles) -> {
+      // Rollbacks deletes instants from timeline. The instant being rolled-back may not have been synced to the
+      // metadata table. Hence, the deleted filed need to be checked against the metadata.
+      try {
+        int origCount = deletedFiles.size();
+        FileStatus[] existingStatuses = getAllFilesInPartition(new Path(basePath, partition));
+        Set<String> currentFiles =
+            Arrays.stream(existingStatuses).map(s -> s.getPath().getName()).collect(Collectors.toSet());
+        deletedFiles.removeIf(f -> !currentFiles.contains(f));
+        if (deletedFiles.size() != origCount) {
+          LOG.warn("Some Files to be deleted as part of " + operation + " at " + instantTime + " were not found in the "
+              + " metadata for partition " + partition
+              + ". To delete = " + origCount + ", found=" + deletedFiles.size());
+        }
+
+        // Commit this record even if there are no deleted files. This will create a delta-commit corresponding to
+        // the instant being synced.
+        fileDeleteCount[0] += deletedFiles.size();
+
+        // Files deleted from a partition
+        HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+            Option.of(new ArrayList<>(deletedFiles)));
+        records.add(record);
+
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to commit rollback deletes at instant " + instantTime, e);
+      }
+    });
+
+    if (!records.isEmpty()) {
+      LOG.info("Updating at " + instantTime + " from " + operation + ". #partitions_updated=" + records.size()
+          + ", #files_deleted=" + fileDeleteCount);
+      commit(records, instantTime, true);
+    }
+  }
+
+  /**
+   * Commit the {@code HoodieRecord}s to Metadata Table as a new delta-commit.
+   *
+   * @param records The records to commit
+   * @param instantTime The timestamp of instant to create
+   * @param performUpdate If True, records are upserted. If False, records are inserted
+   */
+  private synchronized void commit(List<HoodieRecord> records, String instantTime, boolean performUpdate) {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be committed in readonly mode");
+    ValidationUtils.checkArgument(!records.isEmpty());
+
+    // Close all readers
+    if (basefileReader != null) {
+      basefileReader.close();
+      basefileReader = null;
+    }
+    logRecordScanner = null;
+
+    try (HoodieWriteClient writeClient = new HoodieWriteClient(jsc, config, true)) {
+      writeClient.startCommitWithTime(instantTime);
+      List<WriteStatus> statuses;
+      if (performUpdate) {
+        statuses = writeClient.upsert(jsc.parallelize(records), instantTime).collect();
+      } else {
+        statuses = writeClient.insert(jsc.parallelize(records), instantTime).collect();
+      }
+      statuses.forEach(writeStatus -> {
+        if (writeStatus.hasErrors()) {
+          throw new HoodieMetadataException("Failed to commit metadata table records at instant " + instantTime);
+        }
+      });
+    }
+  }
+
+  static String getMetadataTableBasePath(String tableBasePath) {
+    return tableBasePath + Path.SEPARATOR + METADATA_TABLE_REL_PATH;
+  }
+
+  /**
+   * Returns a list of all partitions.
+   */
+  List<String> getAllPartitionPaths() throws IOException {
+    final Timer.Context context = this.metrics.getMetadataCtx(LOOKUP_PARTITIONS_STR);
+
+    Option<HoodieRecord<HoodieMetadataPayload>> hoodieRecord = getMergedRecordByKey(HoodieMetadataImpl.RECORDKEY_PARTITION_LIST);
+    List<String> partitions = Collections.emptyList();
+    if (hoodieRecord.isPresent()) {
+      if (!hoodieRecord.get().getData().getDeletions().isEmpty()) {
+        throw new HoodieMetadataException("Metadata partition list record is inconsistent: "
+            + hoodieRecord.get().getData());
+      }
+
+      partitions = hoodieRecord.get().getData().getFilenames();
+    }
+
+    ++numLookupsAllPartitions;
+
+    long validateTime = 0;
+    if (validateLookups) {
+      // Validate the Metadata Table data by listing the partitions from the file system
+      final Timer.Context contextValidate = this.metrics.getMetadataCtx(VALIDATE_PARTITIONS_STR);
+      List<String> actualPartitions  = FSUtils.getAllPartitionPaths(metaClient.getFs(), basePath, false);
+
+      Collections.sort(actualPartitions);
+      Collections.sort(partitions);
+      if (!actualPartitions.equals(partitions)) {
+        LOG.error("Validation of metadata partition list failed. Lists do not match.");
+        LOG.error("Partitions from metadata: " + Arrays.toString(partitions.toArray()));
+        LOG.error("Partitions from file system: " + Arrays.toString(actualPartitions.toArray()));
+
+        ++numValidationErrors;
+        metrics.updateMetadataMetrics(VALIDATE_ERRORS_STR, 0, numValidationErrors);
+      }
+
+      // Return the direct listing as it should be correct
+      partitions = actualPartitions;
+
+      if (contextValidate != null) {
+        validateTime = metrics.getDurationInMs(contextValidate.stop());
+        metrics.updateMetadataMetrics(VALIDATE_PARTITIONS_STR, validateTime, numLookupsAllPartitions);
+      }
+    }
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop()) - validateTime;
+      metrics.updateMetadataMetrics(LOOKUP_PARTITIONS_STR, durationInMs, numLookupsAllPartitions);
+    }
+
+    return partitions;
+  }
+
+  /**
+   * Return all the files from the partition.
+   *
+   * @param partitionPath The absolute path of the partition
+   */
+  FileStatus[] getAllFilesInPartition(Path partitionPath) throws IOException {
+    final Timer.Context context = this.metrics.getMetadataCtx(LOOKUP_FILES_STR);
+
+    String partitionName = FSUtils.getRelativePartitionPath(new Path(basePath), partitionPath);
+
+    Option<HoodieRecord<HoodieMetadataPayload>> hoodieRecord = getMergedRecordByKey(partitionName);
+    FileStatus[] statuses = {};
+    if (hoodieRecord.isPresent()) {
+      if (!hoodieRecord.get().getData().getDeletions().isEmpty()) {
+        throw new HoodieMetadataException("Metadata record for partition " + partitionName + " is inconsistent: "
+              + hoodieRecord.get().getData());
+      }
+      statuses = hoodieRecord.get().getData().getFileStatuses(partitionPath);
+    }
+
+    ++numLookupsFilesInPartition;
+
+    long validateTime = 0;
+    if (validateLookups) {
+      // Validate the Metadata Table data by listing the partitions from the file system
+      final Timer.Context contextValidate = this.metrics.getMetadataCtx(VALIDATE_FILES_STR);
+      validateTime = System.currentTimeMillis();
+      // Compare the statuses from metadata to those from listing directly from the file system.
+      // Ignore partition metadata file
+      FileStatus[] directStatuses = metaClient.getFs().listStatus(partitionPath,
+          p -> !p.getName().equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE));
+      List<String> directFilenames = Arrays.stream(directStatuses)
+          .map(s -> s.getPath().getName()).collect(Collectors.toList());
+      List<String> metadataFilenames = Arrays.stream(statuses)
+          .map(s -> s.getPath().getName()).collect(Collectors.toList());
+
+      Collections.sort(metadataFilenames);
+      Collections.sort(directFilenames);
+      if (!metadataFilenames.equals(directFilenames)) {
+        LOG.error("Validation of metadata file listing for partition " + partitionName + " failed.");
+        LOG.error("File list from metadata: " + Arrays.toString(metadataFilenames.toArray()));
+        LOG.error("File list from direct listing: " + Arrays.toString(directFilenames.toArray()));
+
+        ++numValidationErrors;
+        metrics.updateMetadataMetrics(VALIDATE_ERRORS_STR, 0, numValidationErrors);
+      }
+
+      // Return the direct listing as it should be correct
+      statuses = directStatuses;
+
+      if (contextValidate != null) {
+        validateTime = metrics.getDurationInMs(contextValidate.stop());
+        metrics.updateMetadataMetrics(VALIDATE_FILES_STR, validateTime, numLookupsFilesInPartition);
+      }
+    }
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop()) - validateTime;
+      metrics.updateMetadataMetrics(LOOKUP_FILES_STR, durationInMs, numLookupsFilesInPartition);
+    }
+
+    return statuses;
+  }
+
+  /**
+   * Open readers to the base and log files.
+   */
+  private synchronized void openBaseAndLogFiles() throws IOException {
+    if (logRecordScanner != null) {
+      // Already opened
+      return;
+    }
+
+    final Timer.Context context = this.metrics.getMetadataCtx(SCAN_STR);
+
+    // Find the latest file slice
+    HoodieTimeline timeline = metaClient.reloadActiveTimeline();
+    HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+    List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+    ValidationUtils.checkArgument(latestSlices.size() == 1);
+
+    // If the base file is present then create a reader
+    Option<HoodieBaseFile> basefile = latestSlices.get(0).getBaseFile();
+    if (basefile.isPresent()) {
+      String basefilePath = basefile.get().getPath();
+      basefileReader = HoodieFileReaderFactory.getFileReader(hadoopConf, new Path(basefilePath));
+
+      LOG.info("Opened metadata base file from " + basefilePath + " at instant " + basefile.get().getCommitTime());
+    }
+
+    // Open the log record scanner using the log files from the latest file slice
+    List<String> logFilePaths = latestSlices.get(0).getLogFiles().map(o -> o.getPath().toString())
+        .collect(Collectors.toList());
+    Option<HoodieInstant> lastInstant = timeline.filterCompletedInstants().lastInstant();
+    String latestTimestamp = lastInstant.isPresent() ? lastInstant.get().getTimestamp() : "";
+
+    logRecordScanner =
+        new HoodieMetadataMergedLogRecordScanner(FSUtils.getFs(basePath, hadoopConf), metadataBasePath,
+            logFilePaths, schema, latestTimestamp, maxMemorySizeInBytes, bufferSize,
+            config.getSpillableMapBasePath(), null);
+
+    LOG.info("Opened metadata log files from " + logFilePaths + " at instant " + lastInstant);
+
+    ++numScans;
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      metrics.updateMetadataMetrics(SCAN_STR, durationInMs, numScans);
+    }
+  }
+
+  /**
+   * Retrieve the merged {@code HoodieRecord} mapped to the given key.
+   *
+   * @param key The key of the record
+   */
+  private Option<HoodieRecord<HoodieMetadataPayload>> getMergedRecordByKey(String key) throws IOException {
+    openBaseAndLogFiles();
+
+    // Retrieve record from base file
+    HoodieRecord<HoodieMetadataPayload> hoodieRecord = null;
+    if (basefileReader != null) {
+      Option<GenericRecord> baseRecord = basefileReader.getRecordByKey(key);
+      if (baseRecord.isPresent()) {
+        hoodieRecord = SpillableMapUtils.convertToHoodieRecordPayload((GenericRecord) baseRecord.get(),
+            metaClient.getTableConfig().getPayloadClass());
+      }
+    }
+
+    // Retrieve record from log file
+    Option<HoodieRecord<HoodieMetadataPayload>> logHoodieRecord = logRecordScanner.getRecordByKey(key);
+    if (logHoodieRecord.isPresent()) {
+      if (hoodieRecord != null) {
+        // Merge the payloads
+        HoodieRecordPayload mergedPayload = hoodieRecord.getData().preCombine(logHoodieRecord.get().getData());
+        hoodieRecord = new HoodieRecord(hoodieRecord.getKey(), mergedPayload);
+      } else {
+        hoodieRecord = logHoodieRecord.get();
+      }
+    }
+
+    return Option.ofNullable(hoodieRecord);
+  }
+
+  /**
+   * Return {@code True} if all Instants from the dataset have been synced with the Metadata Table.
+   */
+  boolean isInSync() {
+    // There should not be any instants to sync
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    return instantsToSync.isEmpty();
+  }
+
+  /**
+   * Return the timestamp of the latest compaction instant.
+   */
+  Option<String> getLatestCompactionTimestamp() {
+    HoodieTimeline timeline = metaClient.reloadActiveTimeline();
+    Option<HoodieInstant> lastCompactionInstant = timeline.filterCompletedInstants()
+        .filter(i -> i.getAction().equals(HoodieTimeline.COMMIT_ACTION)).lastInstant();
+
+    if (lastCompactionInstant.isPresent()) {
+      return Option.of(lastCompactionInstant.get().getTimestamp());
+    } else {
+      return Option.empty();
+    }
+  }
+
+  /**
+   * Return {@code True} if the given absolute path contains a Metadata Table.
+   *
+   * @param basePath The absolute path to test
+   */
+  static boolean isMetadataTable(String basePath) {
+    return basePath.contains(METADATA_TABLE_REL_PATH);

Review comment:
       a stricter check using `endsWith` ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-705760186


   @prashantwason if you can take a pass over the last batch of my comments, that would be great. There was nt much there that we could resolve based on prior discussions. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491217748



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java
##########
@@ -244,6 +245,10 @@ private HoodieCleanMetadata runClean(HoodieTable<?> table, HoodieInstant cleanIn
         inflightInstant = cleanInstant;
       }
 
+      // Update Metadata Table before even finishing clean. This ensures that an async clean operation in the
+      // background does not lead to stale metadata being returned from Metadata Table.
+      HoodieMetadata.update(config, cleanerPlan, cleanInstant.getTimestamp());

Review comment:
       I dont have Async tests yet. 
   
   We need to decide where to log update for async operations. If we do it at Async operation completion then we may end up in multi writer scenario. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-694537286


   @prashantwason the tests are exceeding limit/failing, after fixing checkstyle issues also. Can you please check?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491216511



##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -116,6 +119,17 @@
   public static final String MAX_CONSISTENCY_CHECKS_PROP = "hoodie.consistency.check.max_checks";
   public static int DEFAULT_MAX_CONSISTENCY_CHECKS = 7;
 
+  // Enable the internal Metadata Table which saves file listings
+  private static final String USE_FILE_LISTING_METADATA = "hoodie.metadata.enable";

Review comment:
       Renamed to hoodie.metadata.file.listings.enable




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491330774



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)
+            && (metaSyncTimestamp.isEmpty() || HoodieTimeline.compareTimestamps(i.getTimestamp(),
+                HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp)))
+        .collect(Collectors.toMap(HoodieInstant::getTimestamp, Function.identity()));
+
+    List<HoodieInstant> instantsToSync = new LinkedList<>();
+    datasetInstants.forEach(instant -> {
+      if (metadataInstantMap.containsKey(instant.getTimestamp())) {
+        // instant already synced to metadata table
+        if (!instantsToSync.isEmpty()) {
+          LOG.warn("Found out-of-order already synced instant " + instant + ". Instants to sync=" + instantsToSync);
+        }
+      } else {
+        instantsToSync.add(instant);
+      }
+    });
+
+    // TODO: async clean and async compaction are not handled here. They have a timestamp which is in the past
+    // (when the operation was scheduled) and even on completion they retain their old timestamp.
+    return instantsToSync;
+  }
+
+  /**
+   * Update from {@code HoodieCommitMetadata}.
+   *
+   * @param commitMetadata {@code HoodieCommitMetadata}
+   * @param instantTime Timestamp at which the commit was performed
+   */
+  void update(HoodieCommitMetadata commitMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    commitMetadata.getPartitionToWriteStats().forEach((partition, writeStats) -> {
+      Map<String, Long> newFiles = new HashMap<>(writeStats.size());
+      writeStats.forEach(hoodieWriteStat -> {
+        String pathWithPartition = hoodieWriteStat.getPath();
+        if (pathWithPartition == null) {
+          // Empty partition
+          return;
+        }
+        String filename = pathWithPartition.substring(partition.length() + 1);
+        ValidationUtils.checkState(!newFiles.containsKey(filename), "Duplicate files in HoodieCommitMetadata");
+        newFiles.put(filename, hoodieWriteStat.getTotalWriteBytes());
+      });
+
+      // New files added to a partition
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.of(newFiles),
+          Option.empty());
+      records.add(record);
+    });
+
+    // New partitions created
+    HoodieRecord record = HoodieMetadataPayload.createPartitionListRecord(
+        new ArrayList<String>(commitMetadata.getPartitionToWriteStats().keySet()));
+    records.add(record);
+
+    LOG.info("Updating at " + instantTime + " from Commit/" + commitMetadata.getOperationType()
+        + ". #partitions_updated=" + records.size());
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanerPlan}.
+   *
+   * @param cleanerPlan {@code HoodieCleanerPlan}
+   * @param instantTime Timestamp at which the clean plan was generated
+   */
+  void update(HoodieCleanerPlan cleanerPlan, String instantTime) {
+    // TODO: Cleaner update is called even before the operation has completed. Hence, this function may be called
+    // multiple times and we dont need to update metadata again.
+    HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+    long cnt = timeline.filterCompletedInstants().getInstants().filter(i -> i.getTimestamp().equals(instantTime)).count();
+    if (cnt == 1) {
+      LOG.info("Ignoring update from cleaner plan for already completed instant " + instantTime);
+      return;
+    }
+
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+    cleanerPlan.getFilePathsToBeDeletedPerPartition().forEach((partition, deletedPathInfo) -> {
+      fileDeleteCount[0] += deletedPathInfo.size();
+
+      // Files deleted from a partition
+      List<String> deletedFilenames = deletedPathInfo.stream().map(p -> new Path(p.getFilePath()).getName())
+          .collect(Collectors.toList());
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(deletedFilenames));
+      records.add(record);
+    });
+
+    LOG.info("Updating at " + instantTime + " from CleanerPlan. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanMetadata}.
+   *
+   * @param cleanMetadata {@code HoodieCleanMetadata}
+   * @param instantTime Timestamp at which the clean was completed
+   */
+  private void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+
+    cleanMetadata.getPartitionMetadata().forEach((partition, partitionMetadata) -> {
+      // Files deleted from a partition
+      List<String> deletedFiles = partitionMetadata.getSuccessDeleteFiles();
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(new ArrayList<>(deletedFiles)));
+
+      records.add(record);
+      fileDeleteCount[0] += deletedFiles.size();
+    });
+
+    LOG.info("Updating at " + instantTime + " from Clean. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(records, instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieRestoreMetadata}.
+   *
+   * @param restoreMetadata {@code HoodieRestoreMetadata}
+   * @param instantTime Timestamp at which the restore was performed
+   */
+  void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    restoreMetadata.getHoodieRestoreMetadata().values().forEach(rms -> {
+      rms.forEach(rm -> processRollbackMetadata(rm, partitionToDeletedFiles));
+    });
+
+    commitRollbackDeletes(partitionToDeletedFiles, instantTime, "Restore");
+  }
+
+  /**
+   * Update from {@code HoodieRollbackMetadata}.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param instantTime Timestamp at which the rollback was performed
+   */
+  void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    processRollbackMetadata(rollbackMetadata, partitionToDeletedFiles);
+    commitRollbackDeletes(partitionToDeletedFiles, instantTime, "Rollback");
+  }
+
+  /**
+   * Retrives the partition and deleted files from {@code HoodieRollbackMetadata}.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param partitionToDeletedFiles The {@code Map} to fill with partition and deleted files.
+   */
+  private void processRollbackMetadata(HoodieRollbackMetadata rollbackMetadata,
+                                       Map<String, List<String>> partitionToDeletedFiles) {
+    rollbackMetadata.getPartitionMetadata().values().forEach(pm -> {
+      // Extract deleted file name from the absolute paths saved in getSuccessDeleteFiles()
+      List<String> deletedFiles = pm.getSuccessDeleteFiles().stream().map(p -> new Path(p).getName())
+          .collect(Collectors.toList());
+      if (partitionToDeletedFiles.containsKey(pm.getPartitionPath())) {
+        partitionToDeletedFiles.get(pm.getPartitionPath()).addAll(deletedFiles);
+      } else {
+        partitionToDeletedFiles.put(pm.getPartitionPath(), new ArrayList<>(deletedFiles));
+      }
+    });
+  }
+
+  /**
+   * Create file delete records and commit.
+   *
+   * @param partitionToDeletedFiles {@code Map} of partitions and the deleted files
+   * @param instantTime Timestamp at which the deletes took place
+   * @param operation Type of the operation which caused the files to be deleted
+   */
+  private void commitRollbackDeletes(Map<String, List<String>> partitionToDeletedFiles, String instantTime,
+                                     String operation) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+    partitionToDeletedFiles.forEach((partition, deletedFiles) -> {
+      // Rollbacks deletes instants from timeline. The instant being rolled-back may not have been synced to the
+      // metadata table. Hence, the deleted filed need to be checked against the metadata.
+      try {
+        int origCount = deletedFiles.size();
+        FileStatus[] existingStatuses = getAllFilesInPartition(new Path(basePath, partition));
+        Set<String> currentFiles =
+            Arrays.stream(existingStatuses).map(s -> s.getPath().getName()).collect(Collectors.toSet());
+        deletedFiles.removeIf(f -> !currentFiles.contains(f));
+        if (deletedFiles.size() != origCount) {
+          LOG.warn("Some Files to be deleted as part of " + operation + " at " + instantTime + " were not found in the "
+              + " metadata for partition " + partition
+              + ". To delete = " + origCount + ", found=" + deletedFiles.size());
+        }
+
+        // Commit this record even if there are no deleted files. This will create a delta-commit corresponding to
+        // the instant being synced.
+        fileDeleteCount[0] += deletedFiles.size();
+
+        // Files deleted from a partition
+        HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+            Option.of(new ArrayList<>(deletedFiles)));
+        records.add(record);
+
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to commit rollback deletes at instant " + instantTime, e);
+      }
+    });
+
+    if (!records.isEmpty()) {
+      LOG.info("Updating at " + instantTime + " from " + operation + ". #partitions_updated=" + records.size()
+          + ", #files_deleted=" + fileDeleteCount);
+      commit(records, instantTime, true);
+    }
+  }
+
+  /**
+   * Commit the {@code HoodieRecord}s to Metadata Table as a new delta-commit.
+   *
+   * @param records The records to commit
+   * @param instantTime The timestamp of instant to create
+   * @param performUpdate If True, records are upserted. If False, records are inserted
+   */
+  private synchronized void commit(List<HoodieRecord> records, String instantTime, boolean performUpdate) {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be committed in readonly mode");
+    ValidationUtils.checkArgument(!records.isEmpty());
+
+    // Close all readers
+    if (basefileReader != null) {
+      basefileReader.close();
+      basefileReader = null;
+    }
+    logRecordScanner = null;
+
+    try (HoodieWriteClient writeClient = new HoodieWriteClient(jsc, config, true)) {
+      writeClient.startCommitWithTime(instantTime);
+      List<WriteStatus> statuses;
+      if (performUpdate) {
+        statuses = writeClient.upsert(jsc.parallelize(records), instantTime).collect();
+      } else {
+        statuses = writeClient.insert(jsc.parallelize(records), instantTime).collect();
+      }
+      statuses.forEach(writeStatus -> {
+        if (writeStatus.hasErrors()) {
+          throw new HoodieMetadataException("Failed to commit metadata table records at instant " + instantTime);
+        }
+      });
+    }
+  }
+
+  static String getMetadataTableBasePath(String tableBasePath) {
+    return tableBasePath + Path.SEPARATOR + METADATA_TABLE_REL_PATH;
+  }
+
+  /**
+   * Returns a list of all partitions.
+   */
+  List<String> getAllPartitionPaths() throws IOException {
+    final Timer.Context context = this.metrics.getMetadataCtx(LOOKUP_PARTITIONS_STR);
+
+    Option<HoodieRecord<HoodieMetadataPayload>> hoodieRecord = getMergedRecordByKey(HoodieMetadataImpl.RECORDKEY_PARTITION_LIST);
+    List<String> partitions = Collections.emptyList();
+    if (hoodieRecord.isPresent()) {
+      if (!hoodieRecord.get().getData().getDeletions().isEmpty()) {
+        throw new HoodieMetadataException("Metadata partition list record is inconsistent: "
+            + hoodieRecord.get().getData());
+      }
+
+      partitions = hoodieRecord.get().getData().getFilenames();
+    }
+
+    ++numLookupsAllPartitions;
+
+    long validateTime = 0;
+    if (validateLookups) {
+      // Validate the Metadata Table data by listing the partitions from the file system
+      final Timer.Context contextValidate = this.metrics.getMetadataCtx(VALIDATE_PARTITIONS_STR);
+      List<String> actualPartitions  = FSUtils.getAllPartitionPaths(metaClient.getFs(), basePath, false);
+
+      Collections.sort(actualPartitions);
+      Collections.sort(partitions);
+      if (!actualPartitions.equals(partitions)) {
+        LOG.error("Validation of metadata partition list failed. Lists do not match.");
+        LOG.error("Partitions from metadata: " + Arrays.toString(partitions.toArray()));
+        LOG.error("Partitions from file system: " + Arrays.toString(actualPartitions.toArray()));
+
+        ++numValidationErrors;
+        metrics.updateMetadataMetrics(VALIDATE_ERRORS_STR, 0, numValidationErrors);
+      }
+
+      // Return the direct listing as it should be correct
+      partitions = actualPartitions;
+
+      if (contextValidate != null) {
+        validateTime = metrics.getDurationInMs(contextValidate.stop());
+        metrics.updateMetadataMetrics(VALIDATE_PARTITIONS_STR, validateTime, numLookupsAllPartitions);
+      }
+    }
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop()) - validateTime;
+      metrics.updateMetadataMetrics(LOOKUP_PARTITIONS_STR, durationInMs, numLookupsAllPartitions);
+    }
+
+    return partitions;
+  }
+
+  /**
+   * Return all the files from the partition.
+   *
+   * @param partitionPath The absolute path of the partition
+   */
+  FileStatus[] getAllFilesInPartition(Path partitionPath) throws IOException {
+    final Timer.Context context = this.metrics.getMetadataCtx(LOOKUP_FILES_STR);
+
+    String partitionName = FSUtils.getRelativePartitionPath(new Path(basePath), partitionPath);
+
+    Option<HoodieRecord<HoodieMetadataPayload>> hoodieRecord = getMergedRecordByKey(partitionName);
+    FileStatus[] statuses = {};
+    if (hoodieRecord.isPresent()) {
+      if (!hoodieRecord.get().getData().getDeletions().isEmpty()) {
+        throw new HoodieMetadataException("Metadata record for partition " + partitionName + " is inconsistent: "
+              + hoodieRecord.get().getData());
+      }
+      statuses = hoodieRecord.get().getData().getFileStatuses(partitionPath);
+    }
+
+    ++numLookupsFilesInPartition;
+
+    long validateTime = 0;
+    if (validateLookups) {
+      // Validate the Metadata Table data by listing the partitions from the file system
+      final Timer.Context contextValidate = this.metrics.getMetadataCtx(VALIDATE_FILES_STR);
+      validateTime = System.currentTimeMillis();
+      // Compare the statuses from metadata to those from listing directly from the file system.
+      // Ignore partition metadata file
+      FileStatus[] directStatuses = metaClient.getFs().listStatus(partitionPath,
+          p -> !p.getName().equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE));
+      List<String> directFilenames = Arrays.stream(directStatuses)
+          .map(s -> s.getPath().getName()).collect(Collectors.toList());
+      List<String> metadataFilenames = Arrays.stream(statuses)
+          .map(s -> s.getPath().getName()).collect(Collectors.toList());
+
+      Collections.sort(metadataFilenames);
+      Collections.sort(directFilenames);
+      if (!metadataFilenames.equals(directFilenames)) {
+        LOG.error("Validation of metadata file listing for partition " + partitionName + " failed.");
+        LOG.error("File list from metadata: " + Arrays.toString(metadataFilenames.toArray()));
+        LOG.error("File list from direct listing: " + Arrays.toString(directFilenames.toArray()));
+
+        ++numValidationErrors;
+        metrics.updateMetadataMetrics(VALIDATE_ERRORS_STR, 0, numValidationErrors);
+      }
+
+      // Return the direct listing as it should be correct
+      statuses = directStatuses;
+
+      if (contextValidate != null) {
+        validateTime = metrics.getDurationInMs(contextValidate.stop());
+        metrics.updateMetadataMetrics(VALIDATE_FILES_STR, validateTime, numLookupsFilesInPartition);
+      }
+    }
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop()) - validateTime;
+      metrics.updateMetadataMetrics(LOOKUP_FILES_STR, durationInMs, numLookupsFilesInPartition);
+    }
+
+    return statuses;
+  }
+
+  /**
+   * Open readers to the base and log files.
+   */
+  private synchronized void openBaseAndLogFiles() throws IOException {
+    if (logRecordScanner != null) {
+      // Already opened
+      return;
+    }
+
+    final Timer.Context context = this.metrics.getMetadataCtx(SCAN_STR);
+
+    // Find the latest file slice
+    HoodieTimeline timeline = metaClient.reloadActiveTimeline();
+    HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+    List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+    ValidationUtils.checkArgument(latestSlices.size() == 1);
+
+    // If the base file is present then create a reader
+    Option<HoodieBaseFile> basefile = latestSlices.get(0).getBaseFile();
+    if (basefile.isPresent()) {
+      String basefilePath = basefile.get().getPath();
+      basefileReader = HoodieFileReaderFactory.getFileReader(hadoopConf, new Path(basefilePath));
+
+      LOG.info("Opened metadata base file from " + basefilePath + " at instant " + basefile.get().getCommitTime());
+    }
+
+    // Open the log record scanner using the log files from the latest file slice
+    List<String> logFilePaths = latestSlices.get(0).getLogFiles().map(o -> o.getPath().toString())
+        .collect(Collectors.toList());
+    Option<HoodieInstant> lastInstant = timeline.filterCompletedInstants().lastInstant();
+    String latestTimestamp = lastInstant.isPresent() ? lastInstant.get().getTimestamp() : "";
+
+    logRecordScanner =
+        new HoodieMetadataMergedLogRecordScanner(FSUtils.getFs(basePath, hadoopConf), metadataBasePath,
+            logFilePaths, schema, latestTimestamp, maxMemorySizeInBytes, bufferSize,
+            config.getSpillableMapBasePath(), null);
+
+    LOG.info("Opened metadata log files from " + logFilePaths + " at instant " + lastInstant);
+
+    ++numScans;
+
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      metrics.updateMetadataMetrics(SCAN_STR, durationInMs, numScans);
+    }
+  }
+
+  /**
+   * Retrieve the merged {@code HoodieRecord} mapped to the given key.
+   *
+   * @param key The key of the record
+   */
+  private Option<HoodieRecord<HoodieMetadataPayload>> getMergedRecordByKey(String key) throws IOException {
+    openBaseAndLogFiles();
+
+    // Retrieve record from base file
+    HoodieRecord<HoodieMetadataPayload> hoodieRecord = null;
+    if (basefileReader != null) {
+      Option<GenericRecord> baseRecord = basefileReader.getRecordByKey(key);
+      if (baseRecord.isPresent()) {
+        hoodieRecord = SpillableMapUtils.convertToHoodieRecordPayload((GenericRecord) baseRecord.get(),
+            metaClient.getTableConfig().getPayloadClass());
+      }
+    }
+
+    // Retrieve record from log file
+    Option<HoodieRecord<HoodieMetadataPayload>> logHoodieRecord = logRecordScanner.getRecordByKey(key);
+    if (logHoodieRecord.isPresent()) {
+      if (hoodieRecord != null) {
+        // Merge the payloads
+        HoodieRecordPayload mergedPayload = hoodieRecord.getData().preCombine(logHoodieRecord.get().getData());
+        hoodieRecord = new HoodieRecord(hoodieRecord.getKey(), mergedPayload);
+      } else {
+        hoodieRecord = logHoodieRecord.get();
+      }
+    }
+
+    return Option.ofNullable(hoodieRecord);
+  }
+
+  /**
+   * Return {@code True} if all Instants from the dataset have been synced with the Metadata Table.
+   */
+  boolean isInSync() {
+    // There should not be any instants to sync
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    return instantsToSync.isEmpty();
+  }
+
+  /**
+   * Return the timestamp of the latest compaction instant.
+   */
+  Option<String> getLatestCompactionTimestamp() {
+    HoodieTimeline timeline = metaClient.reloadActiveTimeline();
+    Option<HoodieInstant> lastCompactionInstant = timeline.filterCompletedInstants()
+        .filter(i -> i.getAction().equals(HoodieTimeline.COMMIT_ACTION)).lastInstant();
+
+    if (lastCompactionInstant.isPresent()) {
+      return Option.of(lastCompactionInstant.get().getTimestamp());
+    } else {
+      return Option.empty();
+    }
+  }
+
+  /**
+   * Return {@code True} if the given absolute path contains a Metadata Table.
+   *
+   * @param basePath The absolute path to test
+   */
+  static boolean isMetadataTable(String basePath) {
+    return basePath.contains(METADATA_TABLE_REL_PATH);

Review comment:
       Better. Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [ ] 1. Support for rollbacks in MOR Table
   - [ ] 2. Rollback of metadata if commit eventually fails on dataset 
   - [ x ] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Query-side use of metadata table


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491331857



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * This is a payload which saves information about a single entry in the Metadata Table. The type of the entry is
+ * determined by the "type" saved within the record. The following types of entries are saved:
+ *
+ *   1. List of partitions: There is a single such record
+ *         key="__all_partitions__"
+ *         filenameToSizeMap={"2020/01/01": 0, "2020/01/02": 0, ...}
+ *
+ *   2. List of files in a Partition: There is one such record for each partition
+ *         key=Partition name
+ *         filenameToSizeMap={"file1.parquet": 12345, "file2.parquet": 56789, "file1.log": 9876,
+ *                            "file0.parquet": -1, ...}
+ *
+ *      For deleted files, -1 is used as the size.
+ *
+ *  During compaction on the table, the deletions are merged with additions and hence pruned.
+ */
+public class HoodieMetadataPayload implements HoodieRecordPayload<HoodieMetadataPayload> {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataPayload.class);
+
+  // Represents the size stored for a deleted file
+  private static final long DELETED_FILE_SIZE = -1;
+
+  // Key and type for the metadata record
+  private final String metadataKey;
+  private final PayloadType type;
+
+  // Filenames which are part of this record
+  // key=filename, value=file size (or DELETED_FILE_SIZE to represent a deleted file)
+  private final Map<String, Long> filenameMap = new HashMap<>();
+
+  // Type of the metadata record
+  public enum PayloadType {
+    PARTITION_LIST(1),        // list of partitions
+    PARTITION_FILES(2);       // list of files in a partition
+
+    private final int value;

Review comment:
       That and the fact that I want to avoid 0. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r501943042



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieMergedLogRecordScanner.java
##########
@@ -57,41 +57,58 @@
   private static final Logger LOG = LogManager.getLogger(HoodieMergedLogRecordScanner.class);
 
   // Final map of compacted/merged records
-  private final ExternalSpillableMap<String, HoodieRecord<? extends HoodieRecordPayload>> records;
+  protected final ExternalSpillableMap<String, HoodieRecord<? extends HoodieRecordPayload>> records;
 
   // count of merged records in log
   private long numMergedRecordsInLog;
-
+  private long maxMemorySizeInBytes;
   // Stores the total time taken to perform reading and merging of log blocks
-  private final long totalTimeTakenToReadAndMergeBlocks;
+  private long totalTimeTakenToReadAndMergeBlocks;
   // A timer for calculating elapsed time in millis
   public final HoodieTimer timer = new HoodieTimer();
 
   @SuppressWarnings("unchecked")
   public HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<String> logFilePaths, Schema readerSchema,
       String latestInstantTime, Long maxMemorySizeInBytes, boolean readBlocksLazily, boolean reverseReader,
       int bufferSize, String spillableMapBasePath) {
+    this(fs, basePath, logFilePaths, readerSchema, latestInstantTime, maxMemorySizeInBytes, readBlocksLazily,
+        reverseReader, bufferSize, spillableMapBasePath, true);
+  }
+
+  @SuppressWarnings("unchecked")
+  public HoodieMergedLogRecordScanner(FileSystem fs, String basePath, List<String> logFilePaths, Schema readerSchema,

Review comment:
       moving as comment under HUDI-1319




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491329383



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -194,6 +196,35 @@ public boolean archiveIfRequired(JavaSparkContext jsc) throws IOException {
         .collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
             HoodieInstant.getComparableAction(i.getAction()))));
 
+    // If metadata table is enabled, do not archive instants which are more recent that the latest compaction
+    // of the metadata table. This is required for metadata table sync.
+    if (config.useFileListingMetadata()) {
+      Option<String> latestCompaction = HoodieMetadata.getLatestCompactionTimestamp(config.getBasePath());
+      if (latestCompaction.isPresent()) {
+        LOG.info("Limiting archiving of instants to last compaction on metadata table at " + latestCompaction.get());
+        instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
+            latestCompaction.get()));
+      } else {
+        LOG.info("Not arching instants as there is no compaction yet of the metadata table");

Review comment:
       Fixed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497097206



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)
+            && (metaSyncTimestamp.isEmpty() || HoodieTimeline.compareTimestamps(i.getTimestamp(),
+                HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp)))
+        .collect(Collectors.toMap(HoodieInstant::getTimestamp, Function.identity()));
+
+    List<HoodieInstant> instantsToSync = new LinkedList<>();
+    datasetInstants.forEach(instant -> {
+      if (metadataInstantMap.containsKey(instant.getTimestamp())) {
+        // instant already synced to metadata table
+        if (!instantsToSync.isEmpty()) {
+          LOG.warn("Found out-of-order already synced instant " + instant + ". Instants to sync=" + instantsToSync);
+        }
+      } else {
+        instantsToSync.add(instant);
+      }
+    });
+
+    // TODO: async clean and async compaction are not handled here. They have a timestamp which is in the past
+    // (when the operation was scheduled) and even on completion they retain their old timestamp.
+    return instantsToSync;
+  }
+
+  /**
+   * Update from {@code HoodieCommitMetadata}.
+   *
+   * @param commitMetadata {@code HoodieCommitMetadata}
+   * @param instantTime Timestamp at which the commit was performed
+   */
+  void update(HoodieCommitMetadata commitMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    commitMetadata.getPartitionToWriteStats().forEach((partition, writeStats) -> {
+      Map<String, Long> newFiles = new HashMap<>(writeStats.size());
+      writeStats.forEach(hoodieWriteStat -> {
+        String pathWithPartition = hoodieWriteStat.getPath();
+        if (pathWithPartition == null) {
+          // Empty partition
+          return;
+        }
+        String filename = pathWithPartition.substring(partition.length() + 1);
+        ValidationUtils.checkState(!newFiles.containsKey(filename), "Duplicate files in HoodieCommitMetadata");
+        newFiles.put(filename, hoodieWriteStat.getTotalWriteBytes());
+      });
+
+      // New files added to a partition
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.of(newFiles),
+          Option.empty());
+      records.add(record);
+    });
+
+    // New partitions created
+    HoodieRecord record = HoodieMetadataPayload.createPartitionListRecord(
+        new ArrayList<String>(commitMetadata.getPartitionToWriteStats().keySet()));
+    records.add(record);
+
+    LOG.info("Updating at " + instantTime + " from Commit/" + commitMetadata.getOperationType()
+        + ". #partitions_updated=" + records.size());
+    commit(prepRecords(records, METADATA_PARTITION_NAME), instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanerPlan}.
+   *
+   * @param cleanerPlan {@code HoodieCleanerPlan}
+   * @param instantTime Timestamp at which the clean plan was generated
+   */
+  void update(HoodieCleanerPlan cleanerPlan, String instantTime) {
+    // TODO: Cleaner update is called even before the operation has completed. Hence, this function may be called
+    // multiple times and we dont need to update metadata again.
+    HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+    long cnt = timeline.filterCompletedInstants().getInstants().filter(i -> i.getTimestamp().equals(instantTime)).count();
+    if (cnt == 1) {
+      LOG.info("Ignoring update from cleaner plan for already completed instant " + instantTime);
+      return;
+    }
+
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+    cleanerPlan.getFilePathsToBeDeletedPerPartition().forEach((partition, deletedPathInfo) -> {
+      fileDeleteCount[0] += deletedPathInfo.size();
+
+      // Files deleted from a partition
+      List<String> deletedFilenames = deletedPathInfo.stream().map(p -> new Path(p.getFilePath()).getName())
+          .collect(Collectors.toList());
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(deletedFilenames));
+      records.add(record);
+    });
+
+    LOG.info("Updating at " + instantTime + " from CleanerPlan. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(prepRecords(records, METADATA_PARTITION_NAME), instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanMetadata}.
+   *
+   * @param cleanMetadata {@code HoodieCleanMetadata}
+   * @param instantTime Timestamp at which the clean was completed
+   */
+  private void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+
+    cleanMetadata.getPartitionMetadata().forEach((partition, partitionMetadata) -> {
+      // Files deleted from a partition
+      List<String> deletedFiles = partitionMetadata.getSuccessDeleteFiles();
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(new ArrayList<>(deletedFiles)));
+
+      records.add(record);
+      fileDeleteCount[0] += deletedFiles.size();
+    });
+
+    LOG.info("Updating at " + instantTime + " from Clean. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(prepRecords(records, METADATA_PARTITION_NAME), instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieRestoreMetadata}.
+   *
+   * @param restoreMetadata {@code HoodieRestoreMetadata}
+   * @param instantTime Timestamp at which the restore was performed
+   */
+  void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
+    Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    restoreMetadata.getHoodieRestoreMetadata().values().forEach(rms -> {
+      rms.forEach(rm -> processRollbackMetadata(rm, partitionToDeletedFiles, partitionToAppendedFiles));
+    });
+
+    commitRollback(partitionToDeletedFiles, partitionToAppendedFiles, instantTime, "Restore");
+  }
+
+  /**
+   * Update from {@code HoodieRollbackMetadata}.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param instantTime Timestamp at which the rollback was performed
+   */
+  void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
+    Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    processRollbackMetadata(rollbackMetadata, partitionToDeletedFiles, partitionToAppendedFiles);
+    commitRollback(partitionToDeletedFiles, partitionToAppendedFiles, instantTime, "Rollback");
+  }
+
+  /**
+   * Extracts information about the deleted and append files from the {@code HoodieRollbackMetadata}.
+   *
+   * During a rollback files may be deleted (COW, MOR) or rollback blocks be appended (MOR only) to files. This
+   * function will extract this change file for each partition.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param partitionToDeletedFiles The {@code Map} to fill with files deleted per partition.
+   * @param partitionToAppendedFiles The {@code Map} to fill with files appended per partition and their sizes.
+   */
+  private void processRollbackMetadata(HoodieRollbackMetadata rollbackMetadata,
+                                       Map<String, List<String>> partitionToDeletedFiles,
+                                       Map<String, Map<String, Long>> partitionToAppendedFiles) {
+    rollbackMetadata.getPartitionMetadata().values().forEach(pm -> {
+      final String partition = pm.getPartitionPath();
+
+      if (!pm.getSuccessDeleteFiles().isEmpty()) {
+        if (!partitionToDeletedFiles.containsKey(partition)) {
+          partitionToDeletedFiles.put(partition, new ArrayList<>());
+        }
+
+        // Extract deleted file name from the absolute paths saved in getSuccessDeleteFiles()
+        List<String> deletedFiles = pm.getSuccessDeleteFiles().stream().map(p -> new Path(p).getName())
+            .collect(Collectors.toList());
+        partitionToDeletedFiles.get(partition).addAll(deletedFiles);
+      }
+
+      if (!pm.getAppendFiles().isEmpty()) {
+        if (!partitionToAppendedFiles.containsKey(partition)) {
+          partitionToAppendedFiles.put(partition, new HashMap<>());
+        }
+
+        // Extract appended file name from the absolute paths saved in getAppendFiles()
+        pm.getAppendFiles().forEach((path, size) -> {
+          partitionToAppendedFiles.get(partition).merge(new Path(path).getName(), size, (oldSize, newSizeCopy) -> {
+            return size + oldSize;
+          });
+        });
+      }
+    });
+  }
+
+  /**
+   * Create file delete records and commit.
+   *
+   * @param partitionToDeletedFiles {@code Map} of partitions and the deleted files
+   * @param instantTime Timestamp at which the deletes took place
+   * @param operation Type of the operation which caused the files to be deleted
+   */
+  private void commitRollback(Map<String, List<String>> partitionToDeletedFiles,
+                              Map<String, Map<String, Long>> partitionToAppendedFiles, String instantTime,
+                              String operation) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileChangeCount = {0, 0}; // deletes, appends
+
+    partitionToDeletedFiles.forEach((partition, deletedFiles) -> {
+      // Rollbacks deletes instants from timeline. The instant being rolled-back may not have been synced to the
+      // metadata table. Hence, the deleted filed need to be checked against the metadata.
+      try {
+        FileStatus[] existingStatuses = getAllFilesInPartition(new Path(datasetBasePath, partition));
+        Set<String> currentFiles =
+            Arrays.stream(existingStatuses).map(s -> s.getPath().getName()).collect(Collectors.toSet());
+
+        int origCount = deletedFiles.size();
+        deletedFiles.removeIf(f -> !currentFiles.contains(f));
+        if (deletedFiles.size() != origCount) {
+          LOG.warn("Some Files to be deleted as part of " + operation + " at " + instantTime + " were not found in the "
+              + " metadata for partition " + partition
+              + ". To delete = " + origCount + ", found=" + deletedFiles.size());
+        }
+
+        fileChangeCount[0] += deletedFiles.size();
+
+        Option<Map<String, Long>> filesAdded = Option.empty();
+        if (partitionToAppendedFiles.containsKey(partition)) {
+          filesAdded = Option.of(partitionToAppendedFiles.remove(partition));
+        }
+
+        HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, filesAdded,
+            Option.of(new ArrayList<>(deletedFiles)));
+        records.add(record);
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to commit rollback deletes at instant " + instantTime, e);
+      }
+    });
+
+    partitionToAppendedFiles.forEach((partition, appendedFileMap) -> {
+      fileChangeCount[1] += appendedFileMap.size();
+
+      // Validate that no appended file has been deleted
+      ValidationUtils.checkState(
+          !appendedFileMap.keySet().removeAll(partitionToDeletedFiles.getOrDefault(partition, Collections.emptyList())),
+            "Rollback file cannot both be appended and deleted");
+
+      // New files added to a partition
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.of(appendedFileMap),
+          Option.empty());
+      records.add(record);
+    });
+
+    LOG.info("Updating at " + instantTime + " from " + operation + ". #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileChangeCount[0] + ", #files_appended=" + fileChangeCount[1]);
+    commit(prepRecords(records, METADATA_PARTITION_NAME), instantTime, true);
+  }
+
+  /**
+   * Commit the {@code HoodieRecord}s to Metadata Table as a new delta-commit.
+   *
+   * @param recordRDD The records to commit
+   * @param instantTime The timestamp of instant to create
+   * @param performUpdate If True, records are upserted. If False, records are inserted
+   */
+  private synchronized void commit(JavaRDD<HoodieRecord> recordRDD, String instantTime, boolean performUpdate) {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be committed in readonly mode");
+    //ValidationUtils.checkArgument(!recordRDD.isEmpty());
+
+    // Close all readers
+    if (basefileReader != null) {
+      basefileReader.close();
+      basefileReader = null;
+    }
+    logRecordScanner = null;
+
+    try (HoodieWriteClient writeClient = new HoodieWriteClient(jsc, config, true)) {
+      writeClient.startCommitWithTime(instantTime);
+      List<WriteStatus> statuses;
+      if (performUpdate) {
+        statuses = writeClient.upsertPreppedRecords(recordRDD, instantTime).collect();
+      } else {
+        statuses = writeClient.insertPreppedRecords(recordRDD, instantTime).collect();
+      }
+      statuses.forEach(writeStatus -> {
+        if (writeStatus.hasErrors()) {
+          throw new HoodieMetadataException("Failed to commit metadata table records at instant " + instantTime);
+        }
+      });
+    }
+  }
+
+  /**
+   * Tag each record with the location.
+   *
+   * Since we only read the latest base file in a partition, we tag the records with the instant time of the latest
+   * base file.
+   */
+  private JavaRDD<HoodieRecord> prepRecords(List<HoodieRecord> records, String partitionName) {
+    HoodieTable table = HoodieTable.create(metaClient, config, hadoopConf);
+    SliceView fsView = table.getSliceView();
+    List<HoodieBaseFile> baseFiles = fsView.getLatestFileSlices(partitionName)
+        .map(s -> s.getBaseFile())
+        .filter(b -> b.isPresent())
+        .map(b -> b.get())
+        .collect(Collectors.toList());
+
+    // All the metadata fits within a single base file
+    if (partitionName.equals(METADATA_PARTITION_NAME)) {
+      if (baseFiles.size() > 1) {
+        throw new HoodieMetadataException("Multiple base files found in metadata partition");
+      }
+    }
+
+    String fileId;
+    String instantTime;
+    if (!baseFiles.isEmpty()) {
+      fileId = baseFiles.get(0).getFileId();
+      instantTime = baseFiles.get(0).getCommitTime();
+    } else {
+      // If there is a log file then we can assume that it has the data
+      List<HoodieLogFile> logFiles = fsView.getLatestFileSlices(HoodieMetadataImpl.METADATA_PARTITION_NAME)
+          .map(s -> s.getLatestLogFile())
+          .filter(b -> b.isPresent())
+          .map(b -> b.get())
+          .collect(Collectors.toList());
+      if (logFiles.isEmpty()) {
+        // No base and log files. All are new inserts
+        return jsc.parallelize(records);
+      }
+
+      fileId = logFiles.get(0).getFileId();
+      instantTime = logFiles.get(0).getBaseCommitTime();
+    }
+
+    return jsc.parallelize(records).map(r -> r.setCurrentLocation(new HoodieRecordLocation(instantTime, fileId)));
+  }
+
+  /**
+   * Returns a list of all partitions.
+   */
+  List<String> getAllPartitionPaths() throws IOException {
+    final Timer.Context context = this.metrics.getMetadataCtx(LOOKUP_PARTITIONS_STR);

Review comment:
       Initialized a HoodieMetrics object with metrics disabled for the read-only case. This should keep the code simple by not requiring if checks everywhere this.metrics is used.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497066694



##########
File path: hudi-client/src/main/resources/metadataSchema.txt
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+    "namespace": "hudi.metadata",
+    "type": "record",
+    "name": "metadata",
+    "fields": [
+        {
+            "name": "key",
+            "type": "string"
+        },
+        {
+            "name": "type",
+            "type": "int",
+            "doc": "Type of the metadata record (refer to HoodieMetadataPayload)"
+        },
+        {   "name": "filenameToSizeMap",
+            "type": {
+                "type": "map",
+                "doc": "Filenames mapped to their sizes",
+                "values": {
+                    "type": "long",
+                    "doc": "Size of this file in bytes or -1 for deleted files"

Review comment:
       Good idea. I am going to change this to record which allows for schema evolution in the future.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491217382



##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -116,6 +119,17 @@
   public static final String MAX_CONSISTENCY_CHECKS_PROP = "hoodie.consistency.check.max_checks";
   public static int DEFAULT_MAX_CONSISTENCY_CHECKS = 7;
 
+  // Enable the internal Metadata Table which saves file listings
+  private static final String USE_FILE_LISTING_METADATA = "hoodie.metadata.enable";
+  private static final String DEFAULT_USE_FILE_LISTING_METADATA = "false";
+
+  // Validate contents of Metadata Table on each access against the actual filesystem
+  private static final String FILE_LISTING_METADATA_VERIFY = "hoodie.metadata.verify";
+  private static final String DEFAULT_FILE_LISTING_METADATA_VERIFY = "false";
+
+  // Serialized compaction config to be used for Metadata Table
+  public static final String HOODIE_METADATA_COMPACTION_CONFIG = "hoodie.metadata.compaction.config";

Review comment:
       Not a constant. This is user provided config just like for dataset compaction.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491216884



##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -116,6 +119,17 @@
   public static final String MAX_CONSISTENCY_CHECKS_PROP = "hoodie.consistency.check.max_checks";
   public static int DEFAULT_MAX_CONSISTENCY_CHECKS = 7;
 
+  // Enable the internal Metadata Table which saves file listings
+  private static final String USE_FILE_LISTING_METADATA = "hoodie.metadata.enable";
+  private static final String DEFAULT_USE_FILE_LISTING_METADATA = "false";
+
+  // Validate contents of Metadata Table on each access against the actual filesystem
+  private static final String FILE_LISTING_METADATA_VERIFY = "hoodie.metadata.verify";

Review comment:
       Renamed to hoodie.metadata.file.listings.verify




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-705760186


   @prashantwason if you can take a pass over the last batch of my comments, that would be great. There was nt much there that we could resolve based on prior discussions. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497070209



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataIndex.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIndexException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.index.HoodieIndexUtils;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import scala.Tuple2;
+
+/**
+ * An Index which is specific to the Hoodie Metadata Table.
+ *
+ * Hoodie Metadata Table saves information in specific named partitions. So the partition and the base file location
+ * during indexing can be determined by looking at the record itself. This speeds up the indexing during updates.
+ */
+public class HoodieMetadataIndex<T extends HoodieMetadataPayload> extends HoodieIndex<T> {

Review comment:
       Removed the HoodieIndex and now using the prepped APIs.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [ ] 1. Support for rollbacks in MOR Table
   - [ ] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [ ] 7. Unit test for rollback of partial commits
   - [ ] 8. Schema evolution strategy for metadata table
   - [ ] 9. Unit test for marker based rollback
   - [ ] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491331420



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataIndex.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIndexException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.index.HoodieIndexUtils;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import scala.Tuple2;
+
+/**
+ * An Index which is specific to the Hoodie Metadata Table.
+ *
+ * Hoodie Metadata Table saves information in specific named partitions. So the partition and the base file location
+ * during indexing can be determined by looking at the record itself. This speeds up the indexing during updates.
+ */
+public class HoodieMetadataIndex<T extends HoodieMetadataPayload> extends HoodieIndex<T> {

Review comment:
       Yes we can as this index is really simplistic. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [ ] 1. Support for rollbacks in MOR Table
   - [ ] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Query-side use of metadata table


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [x] 1. Support for rollbacks in MOR Table
   - [x] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [x] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [x] 7. Unit test for rollback of partial commits
   - [x] 8. Schema evolution strategy for metadata table
   - [x] 9. Unit test for marker based rollback
   - [x] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   - [ ] 13. How we are going to add new metadata partitions in the background, as writers/cleaner/compactors keep running.
   - [ ] 14. Fix the case when the table is non-partitioned
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

bvaradar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-701052206


   @prashantwason : Also Have you been able to run metadata list commands successfully ? Did you encounter this issue : Caused by: java.lang.ClassNotFoundException: org.apache.hudi.org.apache.hadoop.hbase.KeyValue$KeyComparator ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [x] 1. Support for rollbacks in MOR Table
   - [x] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [x] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [x] 7. Unit test for rollback of partial commits
   - [x] 8. Schema evolution strategy for metadata table
   - [x] 9. Unit test for marker based rollback
   - [x] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [x] 12. Fix the case when the table is non-partitioned
   - [ ] 13. Test for Async cases
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason closed pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason closed pull request #2064:
URL: https://github.com/apache/hudi/pull/2064


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497074256



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

bvaradar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-699444753


   I instrumented the code to track writeConfig.useFileListingMetadata. It looks like it is being disabled for a dataset that has the metadata dataset
   
   20/09/26 02:58:11 ERROR HoodieMetadata: writeConfig.useFileListingMetadata() =true
   20/09/26 02:58:14 ERROR HoodieMetadata: writeConfig.useFileListingMetadata() =false
   
   The above line is coming from HoodieMetadata.init() method.  I am unable to test using spark.write() as it seems to fallback to listing 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491341294



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataIndex.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;

Review comment:
       Yes, this needs to be done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [ ] 1. Support for rollbacks in MOR Table
   - [ ] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [ ] 7. Unit test for rollback of partial commits
   - [ ] 8. Schema evolution strategy for metadata table
   - [ ] 9. Unit test for marker based rollback
   - [ ] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-698099305


   @umehrot2  Directly using hudi datasource or delta streamer for testing
   should work too. I haven't testing this yet so please let me know if that
   doesn't work.
   
   Query side changes are not implemented yet. So this is ingestion side
   improvement as of today.
   
   On Wed, Sep 23, 2020 at 7:09 PM Udit Mehrotra <no...@github.com>
   wrote:
   
   > @umehrot2 <https://github.com/umehrot2> I have updated the RFC doc with details
   > on how to test RFC-15
   > <https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+and+Query+Planning+Improvements#RFC15:HUDIFileListingandQueryPlanningImprovements-TestPlan>.
   > Please take a look and let me know if I can help in any way.
   >
   > @prashantwason <https://github.com/prashantwason> I missed your earlier
   > pings on this PR. We will start the testing of this PR with S3. Looking at
   > the testing details you provided, I am a bit confused, and have a couple of
   > questions:
   >
   >    -
   >
   >    Do we need to directly use HoodieWriteClient from spark-shell to be
   >    able to test this ? Can't we directly use hudi datasource or delta
   >    streamer for testing with following options set to true while writing:
   >    hoodie.metadata.file.listings.enable,
   >    hoodie.metadata.file.listings.verify ?
   >    -
   >
   >    Also for testing query performance using spark datasource, spark-sql,
   >    hive and presto I am assuming it will detect that metadata table is
   >    present and automatically use that for getting the list ?
   >
   > Will review the PR this week as well to understand the implementation
   > details.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/hudi/pull/2064#issuecomment-698068512>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AN55SSZN3JHLGZ77YHS3RQDSHKS4NANCNFSM4QVMIMDQ>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491219275



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java
##########
@@ -180,14 +181,14 @@ public CleanPlanner(HoodieTable<T> hoodieTable, HoodieWriteConfig config) {
   }
 
   /**
-   * Scan and list all paritions for cleaning.
+   * Scan and list all partitions for cleaning.
    * @return all partitions paths for the dataset.
    * @throws IOException
    */
   private List<String> getPartitionPathsForFullCleaning() throws IOException {
     // Go to brute force mode of scanning all partitions
-    return FSUtils.getAllPartitionPaths(hoodieTable.getMetaClient().getFs(), hoodieTable.getMetaClient().getBasePath(),
-        config.shouldAssumeDatePartitioning());
+    return HoodieMetadata.getAllPartitionPaths(hoodieTable.getMetaClient().getFs(),

Review comment:
       With flags for various operations there is greater chance of eventual inconsistency - async operations may have created/deleted files which are unknown to metadata yet. 
   
   If for certain operations we really need to skip metadata, it will be cleaner to change the API to reflect that. Example:
      HoodieMetadata.getAllPartitionPaths(...., boolean shouldValidate);     When shouldValidate is true, metadata validation if forced leading to file listing being used to return results.
   
   This way we force all listing operation to use single code path which can be optimized later on.    




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491329056



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -253,6 +254,15 @@ public SyncableFileSystemView getHoodieView() {
     return getViewManager().getFileSystemView(metaClient);
   }
 
+  private SyncableFileSystemView getFileSystemViewInternal(HoodieTimeline timeline) {
+    if (config.useFileListingMetadata()) {
+      FileSystemViewStorageConfig viewConfig = config.getViewStorageConfig();
+      return new HoodieMetadataFileSystemView(metaClient, timeline, viewConfig.isIncrementalTimelineSyncEnabled());
+    } else {
+      return getViewManager().getFileSystemView(metaClient);

Review comment:
       Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491220125



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java
##########
@@ -230,6 +231,9 @@ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMeta
     metadata.addMetadata(HoodieCommitMetadata.SCHEMA_KEY, getSchemaToStoreInCommit());
     metadata.setOperationType(operationType);
 
+    // Update Metadata Table
+    HoodieMetadata.update(config, metadata, instantTime);

Review comment:
       I observed that certain commit paths do not go via the HoodieWriteCLient  (HoodieWriteClient.completeCompaction(), AbstractHoodieClient.commitStats()) as they have been moved into the various executors. Hence, I could not find a single location which would be sufficient to issue the  HoodieMetadata.update.
   
   This is also the reason I could not use a member variable in HoodieWriteClient as you had suggested elsewhere. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r501383913



##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();

Review comment:
       Fixed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r501373801



##########
File path: hudi-client/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -121,6 +122,9 @@ public boolean commitStats(String instantTime, List<HoodieWriteStat> stats, Opti
     metadata.setOperationType(operationType);
 
     try {
+      // Update Metadata Table
+      HoodieMetadata.update(config, metadata, instantTime);

Review comment:
       Created HUDI-1320. Will handle there.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-696430277


   Food for thought: we should also think about how we are going to add new metadata partitions in the background, as writers/cleaner/compactors keep running. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r492405639



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadata.java
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.exception.TableNotFoundException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+/**
+ * Defines the interface through which file listing metadata is created, updated and accessed. This metadata is
+ * saved within an internal Metadata Table.
+ *
+ * If file listing metadata is disabled, the functions default to using listStatus(...) RPC calls to retrieve
+ * file listings from the file system.
+ */
+public class HoodieMetadata {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadata.class);
+
+  // Instances of metadata for each basePath
+  private static Map<String, HoodieMetadataImpl> instances = new HashMap<>();

Review comment:
       I don't know if sharing this object leads to any real perf gains. for the use-case you mention, those are two different hudi tables anyway right.  
   
   yes, we need to pass this around the layers. best way is to stick this into `HoodieTable` just like it has a member variable for `index`. Then you can easily access this across layers. This also has to be made `Serializable`. In general, global static objects, when clearly there is a scope for separating the instances is kind of an anti pattern in Java land. Its okay if you want to defer this to me. Just saying that we will probably have to change this code eventually for these reasons. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());

Review comment:
       we are not filtering for completed compactions right?  in fact, you wont find an instant with `compaction` as action type in compelted, since we make that a commit action eventually. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))

Review comment:
       metaTimeline.getDeltaCommitTimeline().filterCompletedInstants()? instead of writing this by hand

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());

Review comment:
       .firstInstant() instead?

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))

Review comment:
       Correction. I think we will still honor scheduled clean plans. but overall, the point remains the same. we can just reflect completed cleans IMO. The only side effect would be that when cleaning by number of retained versions, it can lead to less aggressive cleaning in rare cases. for cleaning by commits, it should be fine, as the cleaning is decided based on a watermark commit time  and file slices before that are discarded. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)

Review comment:
       this can be rewritten using `findInstantsInRange()`

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1003 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Base path of the Metadata Table relative to the dataset (.hoodie/metadata)
+  private static final String METADATA_TABLE_REL_PATH = HoodieTableMetaClient.METAFOLDER_NAME + Path.SEPARATOR
+      + "metadata";
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String basePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String basePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.basePath = basePath;
+    this.metadataBasePath = getMetadataTableBasePath(basePath);
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1)
+        .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexClass(HoodieMetadataIndex.class.getName()).build())
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig().orElseGet(() -> getDefaultCompactionConfig()));
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  private HoodieCompactionConfig getDefaultCompactionConfig() {
+    return HoodieCompactionConfig.newBuilder()
+            .withAutoClean(true)
+            .withInlineCompaction(true)
+            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_COMMITS)
+            .archiveCommitsWith(24, 30)
+            .withMaxNumDeltaCommitsBeforeCompaction(24)
+            .build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, basePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      long totalBaseFileSizeInBytes = 0;
+      long totalLogFileSizeInBytes = 0;
+      int baseFileCount = 0;
+      int logFileCount = 0;
+      HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, metaClient.getActiveTimeline());
+      List<FileSlice> latestSlices = fsView.getLatestFileSlices(METADATA_PARTITION_NAME).collect(Collectors.toList());
+
+      for (FileSlice slice : latestSlices) {
+        if (slice.getBaseFile().isPresent()) {
+          totalBaseFileSizeInBytes += fs.getFileStatus(new Path(slice.getBaseFile().get().getPath())).getLen();
+          ++baseFileCount;
+        }
+        Iterator<HoodieLogFile> it = slice.getLogFiles().iterator();
+        while (it.hasNext()) {
+          totalLogFileSizeInBytes += fs.getFileStatus(it.next().getPath()).getLen();
+          ++logFileCount;
+        }
+      }
+
+      metrics.updateMetadataSizeMetrics(totalBaseFileSizeInBytes, totalLogFileSizeInBytes, baseFileCount, logFileCount);
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(basePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, basePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = basePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(basePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))

Review comment:
       See my note on reflecting compaction after its completed. It has some issues for queries. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497074074



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))

Review comment:
       Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491329825



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -194,6 +196,35 @@ public boolean archiveIfRequired(JavaSparkContext jsc) throws IOException {
         .collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
             HoodieInstant.getComparableAction(i.getAction()))));
 
+    // If metadata table is enabled, do not archive instants which are more recent that the latest compaction
+    // of the metadata table. This is required for metadata table sync.
+    if (config.useFileListingMetadata()) {
+      Option<String> latestCompaction = HoodieMetadata.getLatestCompactionTimestamp(config.getBasePath());
+      if (latestCompaction.isPresent()) {
+        LOG.info("Limiting archiving of instants to last compaction on metadata table at " + latestCompaction.get());
+        instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
+            latestCompaction.get()));
+      } else {
+        LOG.info("Not arching instants as there is no compaction yet of the metadata table");
+        instants = instants.filter(i -> false);
+      }
+    }
+
+    // For metadata tables, ensure commits >= latest compaction commit are retained. This is required for
+    // metadata table sync.
+    if (HoodieMetadata.isMetadataTable(config.getBasePath())) {
+      Option<HoodieInstant> latestCompactionInstant =
+          table.getActiveTimeline().filterPendingCompactionTimeline().lastInstant();
+      if (latestCompactionInstant.isPresent()) {
+        LOG.info("Limiting archiving of instants on metadata table to last compaction at " + latestCompactionInstant.get());
+        instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
+            latestCompactionInstant.get().getTimestamp()));
+      } else {
+        LOG.info("Not archiving instants on metdata table as there is no compaction yet");
+        instants = instants.filter(i -> false);

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497095299



##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();

Review comment:
       The MetaCommand code assumes that you will call "metadata init" first. This is to ensure that listPartitions can be performed both with and without the metadata being initialized for performance comparison purpose.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497072251



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1064 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());

Review comment:
       Yes, HoodieTimeline.COMPACTION_ACTION is simply for completion purpose. Do you propose I remove this or add an exception or code comment?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

bvaradar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497148269



##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+
+    StringBuffer out = new StringBuffer("\n");
+    if (!HoodieMetadata.exists(HoodieCLI.basePath)) {
+      out.append("=== Metadata Table not initilized. Using file listing to get list of partitions. ===\n\n");
+    }
+
+    long t1 = System.currentTimeMillis();
+    List<String> partitions = HoodieMetadata.getAllPartitionPaths(HoodieCLI.fs, HoodieCLI.basePath, false);
+    long t2 = System.currentTimeMillis();
+
+    int[] count = {0};
+    partitions.stream().sorted((p1, p2) -> p2.compareTo(p1)).forEach(p -> {
+      out.append(p);
+      if (++count[0] % 15 == 0) {
+        out.append("\n");
+      } else {
+        out.append(", ");
+      }
+    });
+
+    out.append(String.format("\n\n=== List of partitions retrieved in %.2fsec ===", (t2 - t1) / 1000.0));
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-files", help = "Print a list of all files in a partition from the metadata")
+  public String listFiles(
+      @CliOption(key = {"partition"}, help = "Name of the partition to list files", mandatory = true)
+      final String partition) throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();

Review comment:
       In my case, the metadata is already built first time when I did the bulk-insert. Is it safe to run  metadata init again in this case ?  Can you make the list-files work independently like other show CLI commands ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   1. Support for rollbacks in MOR Table
   2. Rollback of metadata if commit eventually fails on dataset 
   3. HUDI-CLI extensions for metadata debugging
   4. Query-side use of metadata table


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497746566



##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();

Review comment:
       > Adding a note as part of testing : I am not able to query the metadata table after performing a simple bulk-insert on a non-partitioned dataset.
   > 
   
   I tested the partition-less case on spark-shell. It does not work with current code. 
   
   The issue seems to be that in the partition-less case, the name of the partition saved within the Metadata Table is "" (empty) so the FileStatus returned from the metadata table listings are not correct absolute path (first character of file name is missing). I will add a task to fix this case. 
   
   @bvaradar Can you please test with a partitioned case? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491324695



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
##########
@@ -202,23 +202,21 @@ public void scan() {
                     LOG.info("Rolling back the last corrupted log block read in " + logFile.getPath());
                     currentInstantLogBlocks.pop();
                     numBlocksRolledBack++;
-                  } else if (lastBlock.getBlockType() != CORRUPT_BLOCK

Review comment:
       I have simplified the if-else conditions. 
   
   if checks for (lastBlock.getBlockType() == CORRUPT_BLOCK) so the else does not need to check for the != part (which should be true if we reach the else part.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [x] 1. Support for rollbacks in MOR Table
   - [ ] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [x] 7. Unit test for rollback of partial commits
   - [ ] 8. Schema evolution strategy for metadata table
   - [x] 9. Unit test for marker based rollback
   - [x] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   - [ ] 13. How we are going to add new metadata partitions in the background, as writers/cleaner/compactors keep running.
   - [ ] 14. Fix the case when the table is non-partitioned
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r501375567



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -174,8 +174,17 @@ public static String getRelativePartitionPath(Path basePath, Path fullPartitionP
   /**
    * Obtain all the partition paths, that are present in this table, denoted by presence of
    * {@link HoodiePartitionMetadata#HOODIE_PARTITION_METAFILE}.
+   *
+   * If thee basePathStr is a subdirectory of .hoodie folder then we assume that the partitions of an internal
+   * table (a hoodie table within the .hoodie directory) are to be obtained.
+   *
+   * @param fs FileSystem instance
+   * @param basePathStr base directory
    */
   public static List<String> getAllFoldersWithPartitionMetaFile(FileSystem fs, String basePathStr) throws IOException {
+    // If the basePathStr is a folder within the .hoodie directory then we are listing partitions within an
+    // internal table.
+    final boolean isInternalTable = basePathStr.contains(HoodieTableMetaClient.METAFOLDER_NAME);

Review comment:
       Done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491329762



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -194,6 +196,35 @@ public boolean archiveIfRequired(JavaSparkContext jsc) throws IOException {
         .collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
             HoodieInstant.getComparableAction(i.getAction()))));
 
+    // If metadata table is enabled, do not archive instants which are more recent that the latest compaction
+    // of the metadata table. This is required for metadata table sync.
+    if (config.useFileListingMetadata()) {
+      Option<String> latestCompaction = HoodieMetadata.getLatestCompactionTimestamp(config.getBasePath());
+      if (latestCompaction.isPresent()) {
+        LOG.info("Limiting archiving of instants to last compaction on metadata table at " + latestCompaction.get());
+        instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
+            latestCompaction.get()));
+      } else {
+        LOG.info("Not arching instants as there is no compaction yet of the metadata table");
+        instants = instants.filter(i -> false);

Review comment:
       Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491315758



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadata.java
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.exception.TableNotFoundException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+/**
+ * Defines the interface through which file listing metadata is created, updated and accessed. This metadata is
+ * saved within an internal Metadata Table.
+ *
+ * If file listing metadata is disabled, the functions default to using listStatus(...) RPC calls to retrieve
+ * file listings from the file system.
+ */
+public class HoodieMetadata {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadata.class);
+
+  // Instances of metadata for each basePath
+  private static Map<String, HoodieMetadataImpl> instances = new HashMap<>();

Review comment:
       Each time you create an instance of this class, it needs to load the timeline and check for sync, etc. So ideally an instance should be created only once. But the HoodieMetadata is being used in multiple classes (HoodieWriteClient and various XXXCommitActionExectors) so I could not  just keep it at one place.
   
   >> Unless we are doing something here across multiple base paths,
   Yes, we have a usecase in ETL where the same data is written to multiple HUDI tables (cell / row) in parallel. Hence, we need this to be a map.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

bvaradar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r495419947



##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();

Review comment:
       Should there be call to     HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath) ?

##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+
+    StringBuffer out = new StringBuffer("\n");
+    if (!HoodieMetadata.exists(HoodieCLI.basePath)) {
+      out.append("=== Metadata Table not initilized. Using file listing to get list of partitions. ===\n\n");
+    }
+
+    long t1 = System.currentTimeMillis();
+    List<String> partitions = HoodieMetadata.getAllPartitionPaths(HoodieCLI.fs, HoodieCLI.basePath, false);
+    long t2 = System.currentTimeMillis();
+
+    int[] count = {0};
+    partitions.stream().sorted((p1, p2) -> p2.compareTo(p1)).forEach(p -> {
+      out.append(p);
+      if (++count[0] % 15 == 0) {
+        out.append("\n");
+      } else {
+        out.append(", ");
+      }
+    });
+
+    out.append(String.format("\n\n=== List of partitions retrieved in %.2fsec ===", (t2 - t1) / 1000.0));
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-files", help = "Print a list of all files in a partition from the metadata")
+  public String listFiles(
+      @CliOption(key = {"partition"}, help = "Name of the partition to list files", mandatory = true)
+      final String partition) throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();

Review comment:
       Should there be call to     HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath) ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-698099305


   @umehrot2  Directly using hudi datasource or delta streamer for testing
   should work too. I haven't testing this yet so please let me know if that
   doesn't work.
   
   Query side changes are not implemented yet. So this is ingestion side
   improvement as of today.
   
   On Wed, Sep 23, 2020 at 7:09 PM Udit Mehrotra <no...@github.com>
   wrote:
   
   > @umehrot2 <https://github.com/umehrot2> I have updated the RFC doc with details
   > on how to test RFC-15
   > <https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+and+Query+Planning+Improvements#RFC15:HUDIFileListingandQueryPlanningImprovements-TestPlan>.
   > Please take a look and let me know if I can help in any way.
   >
   > @prashantwason <https://github.com/prashantwason> I missed your earlier
   > pings on this PR. We will start the testing of this PR with S3. Looking at
   > the testing details you provided, I am a bit confused, and have a couple of
   > questions:
   >
   >    -
   >
   >    Do we need to directly use HoodieWriteClient from spark-shell to be
   >    able to test this ? Can't we directly use hudi datasource or delta
   >    streamer for testing with following options set to true while writing:
   >    hoodie.metadata.file.listings.enable,
   >    hoodie.metadata.file.listings.verify ?
   >    -
   >
   >    Also for testing query performance using spark datasource, spark-sql,
   >    hive and presto I am assuming it will detect that metadata table is
   >    present and automatically use that for getting the list ?
   >
   > Will review the PR this week as well to understand the implementation
   > details.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/hudi/pull/2064#issuecomment-698068512>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AN55SSZN3JHLGZ77YHS3RQDSHKS4NANCNFSM4QVMIMDQ>
   > .
   >
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [x] 1. Support for rollbacks in MOR Table
   - [ ] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [x] 7. Unit test for rollback of partial commits
   - [ ] 8. Schema evolution strategy for metadata table
   - [x] 9. Unit test for marker based rollback
   - [ ] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   - [ ] 13. How we are going to add new metadata partitions in the background, as writers/cleaner/compactors keep running.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-695076015


   @umehrot2 I have updated the RFC doc with [details on how to test RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+and+Query+Planning+Improvements#RFC15:HUDIFileListingandQueryPlanningImprovements-TestPlan). Please take a look and let me know if I can help in any way.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [x] 1. Support for rollbacks in MOR Table
   - [ ] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [x] 7. Unit test for rollback of partial commits
   - [ ] 8. Schema evolution strategy for metadata table
   - [x] 9. Unit test for marker based rollback
   - [x] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   - [ ] 13. How we are going to add new metadata partitions in the background, as writers/cleaner/compactors keep running.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [x] 1. Support for rollbacks in MOR Table
   - [ ] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [x] 7. Unit test for rollback of partial commits
   - [x] 8. Schema evolution strategy for metadata table
   - [x] 9. Unit test for marker based rollback
   - [x] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   - [ ] 13. How we are going to add new metadata partitions in the background, as writers/cleaner/compactors keep running.
   - [ ] 14. Fix the case when the table is non-partitioned
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-694538029


   Also see a lot of this polluting the test logs. 
   
   ```
   test-trip-table_metadata.timer.deltacommit
                count = 2
            mean rate = 0.00 calls/second
        1-minute rate = 0.00 calls/second
        5-minute rate = 0.00 calls/second
       15-minute rate = 0.07 calls/second
                  min = 702.76 milliseconds
                  max = 1347.67 milliseconds
                 mean = 1017.96 milliseconds
               stddev = 322.38 milliseconds
               median = 702.76 milliseconds
                 75% <= 1347.67 milliseconds
                 95% <= 1347.67 milliseconds
                 98% <= 1347.67 milliseconds
                 99% <= 1347.67 milliseconds
               99.9% <= 1347.67 milliseconds
   ```
   
   can we remove or avoid them on travis


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491313138



##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -884,6 +921,15 @@ public Builder withCompactionConfig(HoodieCompactionConfig compactionConfig) {
       return this;
     }
 
+    public Builder withMetadataCompactionConfig(HoodieCompactionConfig compactionConfig) throws IOException {
+      // Since the property names from HoodieCompactionConfig are already used in withCompactionConfig,

Review comment:
       I feel this will be lots of redundant code especially when we start having other internal tables which will have their own compaction configs. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-701026945


   > I instrumented the code to track writeConfig.useFileListingMetadata. It looks like it is being disabled for a dataset that has the metadata dataset
   > 
   > 20/09/26 02:58:11 ERROR HoodieMetadata: writeConfig.useFileListingMetadata() =true
   > 20/09/26 02:58:14 ERROR HoodieMetadata: writeConfig.useFileListingMetadata() =false
   > 
   > The above line is coming from HoodieMetadata.init() method. I am unable to test using spark.write() as it seems to fallback to listing
   
   
   I think the second log (writeConfig.useFileListingMetadata() =false) is coming from the Hoodie Metadata Table's internal writeConfig. Since the HUDI Metadata Table is also a HUDI Table, it also has a HoodieWriteConfig wherein the useFileListingMetadata is disabled.  
   
   Fallback to listing is by design to prevent issues during adoption.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491312887



##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -755,6 +769,29 @@ public int getBootstrapParallelism() {
     return Integer.parseInt(props.getProperty(HoodieBootstrapConfig.BOOTSTRAP_PARALLELISM));
   }
 
+  /**
+   * File listing metadata configs.
+   */
+  public boolean useFileListingMetadata() {
+    return Boolean.parseBoolean(props.getProperty(USE_FILE_LISTING_METADATA));
+  }
+
+  public boolean getFileListingMetadataVerify() {
+    return Boolean.parseBoolean(props.getProperty(FILE_LISTING_METADATA_VERIFY));
+  }
+
+  public Option<HoodieCompactionConfig> getMetadataCompactionConfig() throws IOException {
+    String serializedCompactionConfig = props.getProperty(HOODIE_METADATA_COMPACTION_CONFIG);
+    if (serializedCompactionConfig == null) {
+      return Option.empty();

Review comment:
       Created a default here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r498587284



##########
File path: hudi-client/src/main/resources/metadataSchema.txt
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+    "namespace": "hudi.metadata",
+    "type": "record",
+    "name": "metadata",
+    "fields": [
+        {
+            "name": "key",
+            "type": "string"
+        },
+        {
+            "name": "type",
+            "type": "int",
+            "doc": "Type of the metadata record (refer to HoodieMetadataPayload)"
+        },
+        {   "name": "filenameToSizeMap",
+            "type": {
+                "type": "map",
+                "doc": "Filenames mapped to their sizes",
+                "values": {
+                    "type": "long",
+                    "doc": "Size of this file in bytes or -1 for deleted files"

Review comment:
       I have changed the schema.

##########
File path: hudi-client/src/main/resources/metadataSchema.txt
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+    "namespace": "hudi.metadata",
+    "type": "record",
+    "name": "metadata",

Review comment:
       I have changed the schema.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] umehrot2 commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

umehrot2 commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-698068512


   > @umehrot2 I have updated the RFC doc with [details on how to test RFC-15](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+and+Query+Planning+Improvements#RFC15:HUDIFileListingandQueryPlanningImprovements-TestPlan). Please take a look and let me know if I can help in any way.
   
   @prashantwason I missed your earlier pings on this PR. We will start the testing of this PR with S3. Looking at the testing details you provided, I am a bit confused, and have a couple of questions:
   
   - Do we need to directly use `HoodieWriteClient` from `spark-shell` to be able to test this ? Can't we directly use `hudi datasource` or `delta streamer` for testing with following options set to true while writing: `hoodie.metadata.file.listings.enable`, `hoodie.metadata.file.listings.verify` ?
   
   - Also for testing query performance using `spark datasource`, `spark-sql`, `hive` and `presto` I am assuming it will detect that metadata table is present and automatically use that for getting the list ?
   
   Will review the PR this week as well to understand the implementation details.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491329056



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -253,6 +254,15 @@ public SyncableFileSystemView getHoodieView() {
     return getViewManager().getFileSystemView(metaClient);
   }
 
+  private SyncableFileSystemView getFileSystemViewInternal(HoodieTimeline timeline) {
+    if (config.useFileListingMetadata()) {
+      FileSystemViewStorageConfig viewConfig = config.getViewStorageConfig();
+      return new HoodieMetadataFileSystemView(metaClient, timeline, viewConfig.isIncrementalTimelineSyncEnabled());
+    } else {
+      return getViewManager().getFileSystemView(metaClient);

Review comment:
       Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491331857



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * This is a payload which saves information about a single entry in the Metadata Table. The type of the entry is
+ * determined by the "type" saved within the record. The following types of entries are saved:
+ *
+ *   1. List of partitions: There is a single such record
+ *         key="__all_partitions__"
+ *         filenameToSizeMap={"2020/01/01": 0, "2020/01/02": 0, ...}
+ *
+ *   2. List of files in a Partition: There is one such record for each partition
+ *         key=Partition name
+ *         filenameToSizeMap={"file1.parquet": 12345, "file2.parquet": 56789, "file1.log": 9876,
+ *                            "file0.parquet": -1, ...}
+ *
+ *      For deleted files, -1 is used as the size.
+ *
+ *  During compaction on the table, the deletions are merged with additions and hence pruned.
+ */
+public class HoodieMetadataPayload implements HoodieRecordPayload<HoodieMetadataPayload> {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataPayload.class);
+
+  // Represents the size stored for a deleted file
+  private static final long DELETED_FILE_SIZE = -1;
+
+  // Key and type for the metadata record
+  private final String metadataKey;
+  private final PayloadType type;
+
+  // Filenames which are part of this record
+  // key=filename, value=file size (or DELETED_FILE_SIZE to represent a deleted file)
+  private final Map<String, Long> filenameMap = new HashMap<>();
+
+  // Type of the metadata record
+  public enum PayloadType {
+    PARTITION_LIST(1),        // list of partitions
+    PARTITION_FILES(2);       // list of files in a partition
+
+    private final int value;

Review comment:
       That and the fact that I want to avoid 0. In my experience 0 is hard to distinguish from not assigned or corrupted. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r484144517



##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -665,6 +669,10 @@ protected void completeCompaction(HoodieCommitMetadata metadata, JavaRDD<WriteSt
     List<HoodieWriteStat> writeStats = writeStatuses.map(WriteStatus::getStat).collect();
     finalizeWrite(table, compactionCommitTime, writeStats);
     LOG.info("Committing Compaction " + compactionCommitTime + ". Finished with result " + metadata);
+
+    // Update Metadata Table
+    HoodieMetadata.update(config, metadata, compactionCommitTime);

Review comment:
       NoTe to self: Where do updates for cleaning etc happen

##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -116,6 +119,17 @@
   public static final String MAX_CONSISTENCY_CHECKS_PROP = "hoodie.consistency.check.max_checks";
   public static int DEFAULT_MAX_CONSISTENCY_CHECKS = 7;
 
+  // Enable the internal Metadata Table which saves file listings
+  private static final String USE_FILE_LISTING_METADATA = "hoodie.metadata.enable";

Review comment:
       Please match the property names with these variables here. 
   
   May be rename this to `hoodie.metadata.file.listings.enable` ? Would be more suitable if you intend to guard file listing alone using this

##########
File path: hudi-client/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -121,6 +122,9 @@ public boolean commitStats(String instantTime, List<HoodieWriteStat> stats, Opti
     metadata.setOperationType(operationType);
 
     try {
+      // Update Metadata Table
+      HoodieMetadata.update(config, metadata, instantTime);

Review comment:
       Would be good to do this from a member variable as opposed to calling a static helper. Will leave more concrete suggestions below

##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -884,6 +921,15 @@ public Builder withCompactionConfig(HoodieCompactionConfig compactionConfig) {
       return this;
     }
 
+    public Builder withMetadataCompactionConfig(HoodieCompactionConfig compactionConfig) throws IOException {
+      // Since the property names from HoodieCompactionConfig are already used in withCompactionConfig,

Review comment:
       Can’t we use the extra name space using `metadata` and store this as individual fields? That’s more extensible and easier to grok IMO

##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java
##########
@@ -244,6 +245,10 @@ private HoodieCleanMetadata runClean(HoodieTable<?> table, HoodieInstant cleanIn
         inflightInstant = cleanInstant;
       }
 
+      // Update Metadata Table before even finishing clean. This ensures that an async clean operation in the
+      // background does not lead to stale metadata being returned from Metadata Table.
+      HoodieMetadata.update(config, cleanerPlan, cleanInstant.getTimestamp());

Review comment:
       Cleaning/Compaction/writing can all log updates in parallel. Do we have a test around this?

##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java
##########
@@ -180,14 +181,14 @@ public CleanPlanner(HoodieTable<T> hoodieTable, HoodieWriteConfig config) {
   }
 
   /**
-   * Scan and list all paritions for cleaning.
+   * Scan and list all partitions for cleaning.
    * @return all partitions paths for the dataset.
    * @throws IOException
    */
   private List<String> getPartitionPathsForFullCleaning() throws IOException {
     // Go to brute force mode of scanning all partitions
-    return FSUtils.getAllPartitionPaths(hoodieTable.getMetaClient().getFs(), hoodieTable.getMetaClient().getBasePath(),
-        config.shouldAssumeDatePartitioning());
+    return HoodieMetadata.getAllPartitionPaths(hoodieTable.getMetaClient().getFs(),

Review comment:
       Should we control this by a flag 

##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java
##########
@@ -230,6 +231,9 @@ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMeta
     metadata.addMetadata(HoodieCommitMetadata.SCHEMA_KEY, getSchemaToStoreInCommit());
     metadata.setOperationType(operationType);
 
+    // Update Metadata Table
+    HoodieMetadata.update(config, metadata, instantTime);

Review comment:
       Note to self: need to understand how this call is different from the call made in writeClient code

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadata.java
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.exception.TableNotFoundException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+/**
+ * Defines the interface through which file listing metadata is created, updated and accessed. This metadata is
+ * saved within an internal Metadata Table.
+ *
+ * If file listing metadata is disabled, the functions default to using listStatus(...) RPC calls to retrieve
+ * file listings from the file system.
+ */
+public class HoodieMetadata {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadata.class);
+
+  // Instances of metadata for each basePath
+  private static Map<String, HoodieMetadataImpl> instances = new HashMap<>();

Review comment:
       Unless we are doing something here across multiple base paths, we can move away from the static map and just deal with one basepath? We can create multiple instances of this class. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -116,6 +119,17 @@
   public static final String MAX_CONSISTENCY_CHECKS_PROP = "hoodie.consistency.check.max_checks";
   public static int DEFAULT_MAX_CONSISTENCY_CHECKS = 7;
 
+  // Enable the internal Metadata Table which saves file listings
+  private static final String USE_FILE_LISTING_METADATA = "hoodie.metadata.enable";
+  private static final String DEFAULT_USE_FILE_LISTING_METADATA = "false";
+
+  // Validate contents of Metadata Table on each access against the actual filesystem
+  private static final String FILE_LISTING_METADATA_VERIFY = "hoodie.metadata.verify";

Review comment:
       Rename to match the property above?

##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -116,6 +119,17 @@
   public static final String MAX_CONSISTENCY_CHECKS_PROP = "hoodie.consistency.check.max_checks";
   public static int DEFAULT_MAX_CONSISTENCY_CHECKS = 7;
 
+  // Enable the internal Metadata Table which saves file listings
+  private static final String USE_FILE_LISTING_METADATA = "hoodie.metadata.enable";
+  private static final String DEFAULT_USE_FILE_LISTING_METADATA = "false";
+
+  // Validate contents of Metadata Table on each access against the actual filesystem
+  private static final String FILE_LISTING_METADATA_VERIFY = "hoodie.metadata.verify";
+  private static final String DEFAULT_FILE_LISTING_METADATA_VERIFY = "false";
+
+  // Serialized compaction config to be used for Metadata Table
+  public static final String HOODIE_METADATA_COMPACTION_CONFIG = "hoodie.metadata.compaction.config";

Review comment:
       If this is just a constant, can we move this else where? This class should just contain user specified configs

##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -755,6 +769,29 @@ public int getBootstrapParallelism() {
     return Integer.parseInt(props.getProperty(HoodieBootstrapConfig.BOOTSTRAP_PARALLELISM));
   }
 
+  /**
+   * File listing metadata configs.
+   */
+  public boolean useFileListingMetadata() {
+    return Boolean.parseBoolean(props.getProperty(USE_FILE_LISTING_METADATA));
+  }
+
+  public boolean getFileListingMetadataVerify() {
+    return Boolean.parseBoolean(props.getProperty(FILE_LISTING_METADATA_VERIFY));
+  }
+
+  public Option<HoodieCompactionConfig> getMetadataCompactionConfig() throws IOException {

Review comment:
       Let’s also move this out?

##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -755,6 +769,29 @@ public int getBootstrapParallelism() {
     return Integer.parseInt(props.getProperty(HoodieBootstrapConfig.BOOTSTRAP_PARALLELISM));
   }
 
+  /**
+   * File listing metadata configs.
+   */
+  public boolean useFileListingMetadata() {
+    return Boolean.parseBoolean(props.getProperty(USE_FILE_LISTING_METADATA));
+  }
+
+  public boolean getFileListingMetadataVerify() {
+    return Boolean.parseBoolean(props.getProperty(FILE_LISTING_METADATA_VERIFY));
+  }
+
+  public Option<HoodieCompactionConfig> getMetadataCompactionConfig() throws IOException {
+    String serializedCompactionConfig = props.getProperty(HOODIE_METADATA_COMPACTION_CONFIG);
+    if (serializedCompactionConfig == null) {
+      return Option.empty();

Review comment:
       If this is indeed a config then define a default value?

##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -174,8 +174,17 @@ public static String getRelativePartitionPath(Path basePath, Path fullPartitionP
   /**
    * Obtain all the partition paths, that are present in this table, denoted by presence of
    * {@link HoodiePartitionMetadata#HOODIE_PARTITION_METAFILE}.
+   *
+   * If thee basePathStr is a subdirectory of .hoodie folder then we assume that the partitions of an internal

Review comment:
       Will look closely at the usage and see if we can simplify this. 

##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataIndex.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;

Review comment:
       These classes that are needed for reading the metadata should probably be in `hudi-common`, so we can use it in query enginesas well

##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -174,8 +174,17 @@ public static String getRelativePartitionPath(Path basePath, Path fullPartitionP
   /**
    * Obtain all the partition paths, that are present in this table, denoted by presence of
    * {@link HoodiePartitionMetadata#HOODIE_PARTITION_METAFILE}.
+   *
+   * If thee basePathStr is a subdirectory of .hoodie folder then we assume that the partitions of an internal

Review comment:
       Typo: thee

##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -253,6 +254,15 @@ public SyncableFileSystemView getHoodieView() {
     return getViewManager().getFileSystemView(metaClient);
   }
 
+  private SyncableFileSystemView getFileSystemViewInternal(HoodieTimeline timeline) {
+    if (config.useFileListingMetadata()) {
+      FileSystemViewStorageConfig viewConfig = config.getViewStorageConfig();
+      return new HoodieMetadataFileSystemView(metaClient, timeline, viewConfig.isIncrementalTimelineSyncEnabled());
+    } else {
+      return getViewManager().getFileSystemView(metaClient);

Review comment:
       Can we make the viewManager hand the metadataFSView as well

##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
##########
@@ -202,23 +202,21 @@ public void scan() {
                     LOG.info("Rolling back the last corrupted log block read in " + logFile.getPath());
                     currentInstantLogBlocks.pop();
                     numBlocksRolledBack++;
-                  } else if (lastBlock.getBlockType() != CORRUPT_BLOCK

Review comment:
       Can you explain what’s changing here in terms of corrupt block handling




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r492405639



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadata.java
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.exception.TableNotFoundException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+/**
+ * Defines the interface through which file listing metadata is created, updated and accessed. This metadata is
+ * saved within an internal Metadata Table.
+ *
+ * If file listing metadata is disabled, the functions default to using listStatus(...) RPC calls to retrieve
+ * file listings from the file system.
+ */
+public class HoodieMetadata {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadata.class);
+
+  // Instances of metadata for each basePath
+  private static Map<String, HoodieMetadataImpl> instances = new HashMap<>();

Review comment:
       I don't know if sharing this object leads to any real perf gains. for the use-case you mention, those are two different hudi tables anyway right.  
   
   yes, we need to pass this around the layers. best way is to stick this into `HoodieTable` just like it has a member variable for `index`. Then you can easily access this across layers. This also has to be made `Serializable`. In general, global static objects, when clearly there is a scope for separating the instances is kind of an anti pattern in Java land. Its okay if you want to defer this to me. Just saying that we will probably have to change this code eventually for these reasons. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-697235280


   >> Can all compaction strategies work off of metadata table itself? Does it have all the data
   
   Yes. Compaction Strategies are based on the sizes of the base and log files involved in the operation or the count of partitions. The metadata table saves the size of each file so compaction strategies should keep working fine. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

bvaradar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r495422383



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataImpl.java
##########
@@ -0,0 +1,1104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieRestoreMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.client.HoodieWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.utils.ClientUtils;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.SpillableMapUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.config.HoodieMetricsConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.io.storage.HoodieFileReader;
+import org.apache.hudi.io.storage.HoodieFileReaderFactory;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import com.codahale.metrics.Timer;
+
+import scala.Tuple2;
+
+/**
+ * Metadata implementation which saves partition and file listing within an internal MOR table
+ * called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync
+ * using the instants on the main dataset.
+ */
+public class HoodieMetadataImpl {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataImpl.class);
+
+  // Table name suffix
+  private static final String METADATA_TABLE_NAME_SUFFIX = "_metadata";
+  // Timestamp for a commit when the base dataset had not had any commits yet.
+  private static final String SOLO_COMMIT_TIMESTAMP = "00000000000000";
+
+  // Name of partition which saves file listings
+  public static final String METADATA_PARTITION_NAME = "metadata_partition";
+  // List of all partitions
+  public static final String[] METADATA_ALL_PARTITIONS = {METADATA_PARTITION_NAME};
+  // Key for the record which saves list of all partitions
+  static final String RECORDKEY_PARTITION_LIST = "__all_partitions__";
+
+  // Metric names
+  private static final String INITIALIZE_STR = "initialize";
+  private static final String SYNC_STR = "sync";
+  private static final String LOOKUP_PARTITIONS_STR = "lookup_partitions";
+  private static final String LOOKUP_FILES_STR = "lookup_files";
+  private static final String VALIDATE_PARTITIONS_STR = "validate_partitions";
+  private static final String VALIDATE_FILES_STR = "validate_files";
+  private static final String VALIDATE_ERRORS_STR = "validate_errors";
+  private static final String SCAN_STR = "scan";
+
+  // Stats names
+  private static final String STAT_TOTAL_BASE_FILE_SIZE = "totalBaseFileSizeInBytes";
+  private static final String STAT_TOTAL_LOG_FILE_SIZE = "totalLogFileSizeInBytes";
+  private static final String STAT_COUNT_BASE_FILES = "baseFileCount";
+  private static final String STAT_COUNT_LOG_FILES = "logFileCount";
+  private static final String STAT_COUNT_PARTITION = "partitionCount";
+  private static final String STAT_IN_SYNC = "isInSync";
+  private static final String STAT_LAST_COMPACTION_TIMESTAMP = "lastCompactionTimestamp";
+
+  private final JavaSparkContext jsc;
+  private final Configuration hadoopConf;
+  private final String datasetBasePath;
+  private final String metadataBasePath;
+  private final HoodieWriteConfig config;
+  private final String tableName;
+  private final Schema schema;
+  private long maxMemorySizeInBytes = 1024 * 1024 * 1024; // TODO
+  private int bufferSize = 10 * 1024 * 1024; // TODO
+  private HoodieTableMetaClient metaClient;
+  private HoodieMetrics metrics;
+  private boolean validateLookups = false;
+  private int numLookupsAllPartitions = 0;
+  private int numLookupsFilesInPartition = 0;
+  private int numScans = 0;
+  private int numValidationErrors = 0;
+  // In read only mode, no changes are made to the metadata table
+  private boolean readOnly = true;
+
+  // Readers for the base and log file which store the metadata
+  private HoodieFileReader<GenericRecord> basefileReader;
+  private HoodieMetadataMergedLogRecordScanner logRecordScanner;
+
+  HoodieMetadataImpl(JavaSparkContext jsc, Configuration hadoopConf, String datasetBasePath, String metadataBasePath,
+                         HoodieWriteConfig writeConfig, boolean readOnly) throws IOException {
+    this.jsc = jsc;
+    this.hadoopConf = hadoopConf;
+    this.datasetBasePath = datasetBasePath;
+    this.metadataBasePath = metadataBasePath;
+    this.readOnly = readOnly;
+
+    this.tableName = writeConfig.getTableName() + METADATA_TABLE_NAME_SUFFIX;
+    this.validateLookups = writeConfig.getFileListingMetadataVerify();
+
+    // Load the schema
+    String schemaStr;
+    try {
+      schemaStr = FileIOUtils.readAsUTFString(getClass().getResourceAsStream("/metadataSchema.txt"));
+      this.schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(schemaStr));
+    } catch (IOException e) {
+      throw new HoodieMetadataException("Failed to read metadata schema", e);
+    }
+
+    if (readOnly) {
+      this.config = null;
+      metaClient = new HoodieTableMetaClient(hadoopConf, metadataBasePath);
+    } else {
+      this.config = createMetadataWriteConfig(writeConfig, schemaStr);
+      this.metrics = new HoodieMetrics(this.config, this.config.getTableName());
+
+      // Inline compaction and auto clean is required as we dont expose this table outside
+      ValidationUtils.checkArgument(this.config.isAutoClean(), "Auto clean is required for Metadata Compaction config");
+      ValidationUtils.checkArgument(this.config.isInlineCompaction(), "Inline compaction is required for Metadata Compaction config");
+      // Metadata Table cannot have its metadata optimized
+      ValidationUtils.checkArgument(this.config.shouldAutoCommit(), "Auto commit is required for Metadata Table");
+      ValidationUtils.checkArgument(!this.config.useFileListingMetadata(), "File listing cannot be used for Metadata Table");
+
+      initAndSyncMetadataTable();
+    }
+  }
+
+  /**
+   * Create a {@code HoodieWriteConfig} to use for the Metadata Table.
+   *
+   * @param writeConfig {@code HoodieWriteConfig} of the main dataset writer
+   * @param schemaStr Metadata Table schema
+   */
+  private HoodieWriteConfig createMetadataWriteConfig(HoodieWriteConfig writeConfig, String schemaStr) throws IOException {
+    // Create the write config for the metadata table by borrowing options from the main write config.
+    HoodieWriteConfig.Builder builder = HoodieWriteConfig.newBuilder()
+        .withTimelineLayoutVersion(writeConfig.getTimelineLayoutVersion())
+        .withConsistencyGuardConfig(ConsistencyGuardConfig.newBuilder()
+            .withConsistencyCheckEnabled(writeConfig.getConsistencyGuardConfig().isConsistencyCheckEnabled())
+            .withInitialConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getInitialConsistencyCheckIntervalMs())
+            .withMaxConsistencyCheckIntervalMs(writeConfig.getConsistencyGuardConfig().getMaxConsistencyCheckIntervalMs())
+            .withMaxConsistencyChecks(writeConfig.getConsistencyGuardConfig().getMaxConsistencyChecks())
+            .build())
+        .withUseFileListingMetadata(false)
+        .withFileListingMetadataVerify(false)
+        .withAutoCommit(true)
+        .withAvroSchemaValidate(true)
+        .withEmbeddedTimelineServerEnabled(false)
+        .withAssumeDatePartitioning(false)
+        .withPath(metadataBasePath)
+        .withSchema(schemaStr)
+        .forTable(tableName)
+        .withParallelism(1, 1).withDeleteParallelism(1).withRollbackParallelism(1).withFinalizeWriteParallelism(1)
+        .withCompactionConfig(writeConfig.getMetadataCompactionConfig());
+
+    if (writeConfig.isMetricsOn()) {
+      HoodieMetricsConfig.Builder metricsConfig = HoodieMetricsConfig.newBuilder()
+          .withReporterType(writeConfig.getMetricsReporterType().toString())
+          .on(true);
+      switch (writeConfig.getMetricsReporterType()) {
+        case GRAPHITE:
+          metricsConfig.onGraphitePort(writeConfig.getGraphiteServerPort())
+                       .toGraphiteHost(writeConfig.getGraphiteServerHost())
+                       .usePrefix(writeConfig.getGraphiteMetricPrefix());
+          break;
+        case JMX:
+          metricsConfig.onJmxPort(writeConfig.getJmxPort())
+                       .toJmxHost(writeConfig.getJmxHost());
+          break;
+        case DATADOG:
+          // TODO:
+          break;
+        case CONSOLE:
+        case INMEMORY:
+          break;
+        default:
+          throw new HoodieMetadataException("Unsupported Metrics Reporter type " + writeConfig.getMetricsReporterType());
+      }
+
+      builder.withMetricsConfig(metricsConfig.build());
+    }
+
+    return builder.build();
+  }
+
+  public HoodieWriteConfig getWriteConfig() {
+    return config;
+  }
+
+  /**
+   * Reload the metadata table by syncing it based on the commits on the dataset.
+   *
+   * This is only allowed
+   */
+
+  public void reload() throws IOException {
+    initAndSyncMetadataTable();
+  }
+
+  /**
+   * Initialize the metadata table if it does not exist. Update the metadata to bring it in sync with the file system.
+   *
+   * This can happen in two ways:
+   * 1. If the metadata table did not exist, then file and partition listing is used
+   * 2. If the metadata table exists, the instants from active timeline are read in order and changes applied
+   *
+   * The above logic has been chosen because it is faster to perform #1 at scale rather than read all the Instants
+   * which are large in size (AVRO or JSON encoded and not compressed) and incur considerable IO for de-serialization
+   * and decoding.
+   */
+  private void initAndSyncMetadataTable() throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    final Timer.Context context = this.metrics.getMetadataCtx(INITIALIZE_STR);
+
+    HoodieTableMetaClient datasetMetaClient = new HoodieTableMetaClient(hadoopConf, datasetBasePath);
+    FileSystem fs = FSUtils.getFs(metadataBasePath, hadoopConf);
+    boolean exists = fs.exists(new Path(metadataBasePath, HoodieTableMetaClient.METAFOLDER_NAME));
+
+    if (!exists) {
+      // Initialize for the first time by listing partitions and files directly from the file system
+      initFromFilesystem(datasetMetaClient);
+    } else {
+      metaClient = ClientUtils.createMetaClient(hadoopConf, config, true);
+    }
+
+    /*
+    // TODO: We may not be able to sync in certain cases (instants archived etc)
+    //if (!canSync(datasetMetaClient)) {
+      // Need to recreate the table as sync has failed
+      // TODO: delete the table
+    //  initFromFilesystem(datasetMetaClient);
+    //}
+    */
+
+    // This is always called even in case the table was created for the first time. This is because
+    // initFromFilesystem() does file listing and hence may take a long time during which some new updates
+    // may have occurred on the table. Hence, calling this always ensures that the metadata is brought in sync
+    // with the active timeline.
+    syncFromInstants(datasetMetaClient);
+
+    // Publish some metrics
+    if (context != null) {
+      long durationInMs = metrics.getDurationInMs(context.stop());
+      // Time to initilize and sync
+      if (exists) {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, 0, 0);
+        metrics.updateMetadataMetrics(SYNC_STR, durationInMs, 1);
+      } else {
+        metrics.updateMetadataMetrics(INITIALIZE_STR, durationInMs, 1);
+        metrics.updateMetadataMetrics(SYNC_STR, 0, 0);
+      }
+
+      // Total size of the metadata and count of base/log files
+      Map<String, String> stats = getStats(false);
+      metrics.updateMetadataSizeMetrics(Long.valueOf(stats.get(STAT_TOTAL_BASE_FILE_SIZE)),
+          Long.valueOf(stats.get(STAT_TOTAL_LOG_FILE_SIZE)), Integer.valueOf(stats.get(STAT_COUNT_BASE_FILES)),
+          Integer.valueOf(stats.get(STAT_COUNT_LOG_FILES)));
+    }
+  }
+
+  /**
+   * Initialize the Metadata Table by listing files and partitions from the file system.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void initFromFilesystem(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be initialized in readonly mode");
+
+    // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+    Option<HoodieInstant> latestInstant = datasetMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+    String createInstantTime = latestInstant.isPresent() ? latestInstant.get().getTimestamp() : SOLO_COMMIT_TIMESTAMP;
+
+    LOG.info("Creating a new metadata table in " + metadataBasePath + " at instant " + createInstantTime);
+    metaClient = HoodieTableMetaClient.initTableType(hadoopConf, metadataBasePath.toString(),
+        HoodieTableType.MERGE_ON_READ, tableName, "archived", HoodieMetadataPayload.class.getName(),
+        HoodieFileFormat.HFILE.toString());
+
+    // List all partitions in the basePath of the containing dataset
+    FileSystem fs = FSUtils.getFs(datasetBasePath, hadoopConf);
+    List<String> partitions = FSUtils.getAllPartitionPaths(fs, datasetBasePath, false);
+    LOG.info("Initializing metadata table by using file listings in " + partitions.size() + " partitions");
+
+    // List all partitions in parallel and collect the files in them
+    final String dbasePath = datasetBasePath;
+    final SerializableConfiguration serializedConf = new SerializableConfiguration(hadoopConf);
+    int parallelism =  Math.min(partitions.size(), 100) + 1; // +1 to ensure non zero
+    JavaPairRDD<String, FileStatus[]> partitionFileListRDD = jsc.parallelize(partitions, parallelism)
+        .mapToPair(partition -> {
+          FileSystem fsys = FSUtils.getFs(dbasePath, serializedConf.get());
+          FileStatus[] statuses = FSUtils.getAllDataFilesInPartition(fsys, new Path(dbasePath, partition));
+          return new Tuple2<>(partition, statuses);
+        });
+
+    // Collect the list of partitions and file lists
+    List<Tuple2<String, FileStatus[]>> partitionFileList = partitionFileListRDD.collect();
+
+    // Create a HoodieCommitMetadata with writeStats for all discovered files
+    int[] stats = {0};
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    partitionFileList.forEach(t -> {
+      final String partition = t._1;
+      try {
+        if (!fs.exists(new Path(datasetBasePath, partition + Path.SEPARATOR + HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE))) {
+          return;
+        }
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to check partition " + partition, e);
+      }
+
+      // If the partition has no files then create a writeStat with no file path
+      if (t._2.length == 0) {
+        HoodieWriteStat writeStat = new HoodieWriteStat();
+        writeStat.setPartitionPath(partition);
+        metadata.addWriteStat(partition, writeStat);
+      } else {
+        Arrays.stream(t._2).forEach(status -> {
+          String filename = status.getPath().getName();
+          if (filename.equals(HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE)) {
+            return;
+          }
+          HoodieWriteStat writeStat = new HoodieWriteStat();
+          writeStat.setPath(partition + Path.SEPARATOR + filename);
+          writeStat.setPartitionPath(partition);
+          writeStat.setTotalWriteBytes(status.getLen());
+          metadata.addWriteStat(partition, writeStat);
+        });
+      }
+      stats[0] += t._2.length;
+    });
+
+    LOG.info("Committing " + partitionFileList.size() + " partitions and " + stats[0] + " files to metadata");
+    update(metadata, createInstantTime);
+  }
+
+  /**
+   * Sync the Metadata Table from the instants created on the dataset.
+   *
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) throws IOException {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be synced in readonly mode");
+
+    List<HoodieInstant> instantsToSync = findInstantsToSync(datasetMetaClient);
+    if (instantsToSync.isEmpty()) {
+      return;
+    }
+
+    LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
+
+    // Read each instant in order and sync it to metadata table
+    final HoodieActiveTimeline timeline = datasetMetaClient.getActiveTimeline();
+    for (HoodieInstant instant : instantsToSync) {
+      LOG.info("Syncing instant " + instant + " to metadata table");
+
+      switch (instant.getAction()) {
+        case HoodieTimeline.CLEAN_ACTION: {
+          // CLEAN is synced from the
+          // - inflight instant which contains the HoodieCleanerPlan, or
+          // - complete instant which contains the HoodieCleanMetadata
+          try {
+            HoodieInstant inflightCleanInstant = new HoodieInstant(true, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(inflightCleanInstant.isInflight());
+            HoodieCleanerPlan cleanerPlan = CleanerUtils.getCleanerPlan(datasetMetaClient, inflightCleanInstant);
+            update(cleanerPlan, instant.getTimestamp());
+          } catch (HoodieIOException e) {
+            HoodieInstant cleanInstant = new HoodieInstant(false, instant.getAction(), instant.getTimestamp());
+            ValidationUtils.checkArgument(cleanInstant.isCompleted());
+            HoodieCleanMetadata cleanMetadata = CleanerUtils.getCleanerMetadata(datasetMetaClient, cleanInstant);
+            update(cleanMetadata, instant.getTimestamp());
+          }
+          break;
+        }
+        case HoodieTimeline.DELTA_COMMIT_ACTION:
+        case HoodieTimeline.COMMIT_ACTION:
+        case HoodieTimeline.COMPACTION_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(
+              timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+          update(commitMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.ROLLBACK_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRollbackMetadata rollbackMetadata = TimelineMetadataUtils.deserializeHoodieRollbackMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(rollbackMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.RESTORE_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          HoodieRestoreMetadata restoreMetadata = TimelineMetadataUtils.deserializeHoodieRestoreMetadata(
+              timeline.getInstantDetails(instant).get());
+          update(restoreMetadata, instant.getTimestamp());
+          break;
+        }
+        case HoodieTimeline.SAVEPOINT_ACTION: {
+          ValidationUtils.checkArgument(instant.isCompleted());
+          // Nothing to be done here
+          break;
+        }
+        default: {
+          throw new HoodieException("Unknown type of action " + instant.getAction());
+        }
+      }
+    }
+  }
+
+  /**
+   * Return an ordered list of instants which have not been synced to the Metadata Table.
+
+   * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+   */
+  public List<HoodieInstant> findInstantsToSync(HoodieTableMetaClient datasetMetaClient) {
+    HoodieTimeline metaTimeline = metaClient.reloadActiveTimeline();
+
+    // All instants since the last time metadata table was compacted are candidates for sync
+    Option<String> compactionTimestamp = getLatestCompactionTimestamp();
+
+    // If there has not been any compaction then the first delta commit instant should be the one at which
+    // the metadata table was created. We should not sync any instants before that creation time.
+    Option<HoodieInstant> oldestMetaInstant = Option.empty();
+    if (!compactionTimestamp.isPresent()) {
+      oldestMetaInstant = Option.fromJavaOptional(metaTimeline.getInstants()
+          .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION))
+          .findFirst());
+      if (oldestMetaInstant.isPresent()) {
+        // TODO: Ensure this is the instant at which we created the metadata table
+      }
+    }
+
+    String metaSyncTimestamp = compactionTimestamp.isPresent() ? compactionTimestamp.get()
+        : oldestMetaInstant.isPresent() ? oldestMetaInstant.get().getTimestamp() : "";
+
+    // Metadata table is updated when an instant is completed except for the following:
+    //  CLEAN: metadata table is updated during inflight. So for CLEAN we accept inflight actions.
+    List<HoodieInstant> datasetInstants = datasetMetaClient.getActiveTimeline().getInstants()
+        .filter(i -> i.isCompleted() || (i.getAction().equals(HoodieTimeline.CLEAN_ACTION) && i.isInflight()))
+        .filter(i -> metaSyncTimestamp.isEmpty()
+            || HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp))
+        .collect(Collectors.toList());
+
+    // Each operation on dataset leads to a delta-commit on the metadata MOR table. So find only delta-commit
+    // instants in metadata table which are after the last compaction.
+    Map<String, HoodieInstant> metadataInstantMap = metaTimeline.getInstants()
+        .filter(i -> i.isCompleted() && i.getAction().equals(HoodieTimeline.DELTA_COMMIT_ACTION)
+            && (metaSyncTimestamp.isEmpty() || HoodieTimeline.compareTimestamps(i.getTimestamp(),
+                HoodieTimeline.GREATER_THAN_OR_EQUALS,
+                metaSyncTimestamp)))
+        .collect(Collectors.toMap(HoodieInstant::getTimestamp, Function.identity()));
+
+    List<HoodieInstant> instantsToSync = new LinkedList<>();
+    datasetInstants.forEach(instant -> {
+      if (metadataInstantMap.containsKey(instant.getTimestamp())) {
+        // instant already synced to metadata table
+        if (!instantsToSync.isEmpty()) {
+          LOG.warn("Found out-of-order already synced instant " + instant + ". Instants to sync=" + instantsToSync);
+        }
+      } else {
+        instantsToSync.add(instant);
+      }
+    });
+
+    // TODO: async clean and async compaction are not handled here. They have a timestamp which is in the past
+    // (when the operation was scheduled) and even on completion they retain their old timestamp.
+    return instantsToSync;
+  }
+
+  /**
+   * Update from {@code HoodieCommitMetadata}.
+   *
+   * @param commitMetadata {@code HoodieCommitMetadata}
+   * @param instantTime Timestamp at which the commit was performed
+   */
+  void update(HoodieCommitMetadata commitMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    commitMetadata.getPartitionToWriteStats().forEach((partition, writeStats) -> {
+      Map<String, Long> newFiles = new HashMap<>(writeStats.size());
+      writeStats.forEach(hoodieWriteStat -> {
+        String pathWithPartition = hoodieWriteStat.getPath();
+        if (pathWithPartition == null) {
+          // Empty partition
+          return;
+        }
+        String filename = pathWithPartition.substring(partition.length() + 1);
+        ValidationUtils.checkState(!newFiles.containsKey(filename), "Duplicate files in HoodieCommitMetadata");
+        newFiles.put(filename, hoodieWriteStat.getTotalWriteBytes());
+      });
+
+      // New files added to a partition
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.of(newFiles),
+          Option.empty());
+      records.add(record);
+    });
+
+    // New partitions created
+    HoodieRecord record = HoodieMetadataPayload.createPartitionListRecord(
+        new ArrayList<String>(commitMetadata.getPartitionToWriteStats().keySet()));
+    records.add(record);
+
+    LOG.info("Updating at " + instantTime + " from Commit/" + commitMetadata.getOperationType()
+        + ". #partitions_updated=" + records.size());
+    commit(prepRecords(records, METADATA_PARTITION_NAME), instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanerPlan}.
+   *
+   * @param cleanerPlan {@code HoodieCleanerPlan}
+   * @param instantTime Timestamp at which the clean plan was generated
+   */
+  void update(HoodieCleanerPlan cleanerPlan, String instantTime) {
+    // TODO: Cleaner update is called even before the operation has completed. Hence, this function may be called
+    // multiple times and we dont need to update metadata again.
+    HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+    long cnt = timeline.filterCompletedInstants().getInstants().filter(i -> i.getTimestamp().equals(instantTime)).count();
+    if (cnt == 1) {
+      LOG.info("Ignoring update from cleaner plan for already completed instant " + instantTime);
+      return;
+    }
+
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+    cleanerPlan.getFilePathsToBeDeletedPerPartition().forEach((partition, deletedPathInfo) -> {
+      fileDeleteCount[0] += deletedPathInfo.size();
+
+      // Files deleted from a partition
+      List<String> deletedFilenames = deletedPathInfo.stream().map(p -> new Path(p.getFilePath()).getName())
+          .collect(Collectors.toList());
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(deletedFilenames));
+      records.add(record);
+    });
+
+    LOG.info("Updating at " + instantTime + " from CleanerPlan. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(prepRecords(records, METADATA_PARTITION_NAME), instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieCleanMetadata}.
+   *
+   * @param cleanMetadata {@code HoodieCleanMetadata}
+   * @param instantTime Timestamp at which the clean was completed
+   */
+  private void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileDeleteCount = {0};
+
+    cleanMetadata.getPartitionMetadata().forEach((partition, partitionMetadata) -> {
+      // Files deleted from a partition
+      List<String> deletedFiles = partitionMetadata.getSuccessDeleteFiles();
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.empty(),
+          Option.of(new ArrayList<>(deletedFiles)));
+
+      records.add(record);
+      fileDeleteCount[0] += deletedFiles.size();
+    });
+
+    LOG.info("Updating at " + instantTime + " from Clean. #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileDeleteCount[0]);
+    commit(prepRecords(records, METADATA_PARTITION_NAME), instantTime, true);
+  }
+
+  /**
+   * Update from {@code HoodieRestoreMetadata}.
+   *
+   * @param restoreMetadata {@code HoodieRestoreMetadata}
+   * @param instantTime Timestamp at which the restore was performed
+   */
+  void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
+    Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    restoreMetadata.getHoodieRestoreMetadata().values().forEach(rms -> {
+      rms.forEach(rm -> processRollbackMetadata(rm, partitionToDeletedFiles, partitionToAppendedFiles));
+    });
+
+    commitRollback(partitionToDeletedFiles, partitionToAppendedFiles, instantTime, "Restore");
+  }
+
+  /**
+   * Update from {@code HoodieRollbackMetadata}.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param instantTime Timestamp at which the rollback was performed
+   */
+  void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
+    Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
+    Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
+    processRollbackMetadata(rollbackMetadata, partitionToDeletedFiles, partitionToAppendedFiles);
+    commitRollback(partitionToDeletedFiles, partitionToAppendedFiles, instantTime, "Rollback");
+  }
+
+  /**
+   * Extracts information about the deleted and append files from the {@code HoodieRollbackMetadata}.
+   *
+   * During a rollback files may be deleted (COW, MOR) or rollback blocks be appended (MOR only) to files. This
+   * function will extract this change file for each partition.
+   *
+   * @param rollbackMetadata {@code HoodieRollbackMetadata}
+   * @param partitionToDeletedFiles The {@code Map} to fill with files deleted per partition.
+   * @param partitionToAppendedFiles The {@code Map} to fill with files appended per partition and their sizes.
+   */
+  private void processRollbackMetadata(HoodieRollbackMetadata rollbackMetadata,
+                                       Map<String, List<String>> partitionToDeletedFiles,
+                                       Map<String, Map<String, Long>> partitionToAppendedFiles) {
+    rollbackMetadata.getPartitionMetadata().values().forEach(pm -> {
+      final String partition = pm.getPartitionPath();
+
+      if (!pm.getSuccessDeleteFiles().isEmpty()) {
+        if (!partitionToDeletedFiles.containsKey(partition)) {
+          partitionToDeletedFiles.put(partition, new ArrayList<>());
+        }
+
+        // Extract deleted file name from the absolute paths saved in getSuccessDeleteFiles()
+        List<String> deletedFiles = pm.getSuccessDeleteFiles().stream().map(p -> new Path(p).getName())
+            .collect(Collectors.toList());
+        partitionToDeletedFiles.get(partition).addAll(deletedFiles);
+      }
+
+      if (!pm.getAppendFiles().isEmpty()) {
+        if (!partitionToAppendedFiles.containsKey(partition)) {
+          partitionToAppendedFiles.put(partition, new HashMap<>());
+        }
+
+        // Extract appended file name from the absolute paths saved in getAppendFiles()
+        pm.getAppendFiles().forEach((path, size) -> {
+          partitionToAppendedFiles.get(partition).merge(new Path(path).getName(), size, (oldSize, newSizeCopy) -> {
+            return size + oldSize;
+          });
+        });
+      }
+    });
+  }
+
+  /**
+   * Create file delete records and commit.
+   *
+   * @param partitionToDeletedFiles {@code Map} of partitions and the deleted files
+   * @param instantTime Timestamp at which the deletes took place
+   * @param operation Type of the operation which caused the files to be deleted
+   */
+  private void commitRollback(Map<String, List<String>> partitionToDeletedFiles,
+                              Map<String, Map<String, Long>> partitionToAppendedFiles, String instantTime,
+                              String operation) {
+    List<HoodieRecord> records = new LinkedList<>();
+    int[] fileChangeCount = {0, 0}; // deletes, appends
+
+    partitionToDeletedFiles.forEach((partition, deletedFiles) -> {
+      // Rollbacks deletes instants from timeline. The instant being rolled-back may not have been synced to the
+      // metadata table. Hence, the deleted filed need to be checked against the metadata.
+      try {
+        FileStatus[] existingStatuses = getAllFilesInPartition(new Path(datasetBasePath, partition));
+        Set<String> currentFiles =
+            Arrays.stream(existingStatuses).map(s -> s.getPath().getName()).collect(Collectors.toSet());
+
+        int origCount = deletedFiles.size();
+        deletedFiles.removeIf(f -> !currentFiles.contains(f));
+        if (deletedFiles.size() != origCount) {
+          LOG.warn("Some Files to be deleted as part of " + operation + " at " + instantTime + " were not found in the "
+              + " metadata for partition " + partition
+              + ". To delete = " + origCount + ", found=" + deletedFiles.size());
+        }
+
+        fileChangeCount[0] += deletedFiles.size();
+
+        Option<Map<String, Long>> filesAdded = Option.empty();
+        if (partitionToAppendedFiles.containsKey(partition)) {
+          filesAdded = Option.of(partitionToAppendedFiles.remove(partition));
+        }
+
+        HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, filesAdded,
+            Option.of(new ArrayList<>(deletedFiles)));
+        records.add(record);
+      } catch (IOException e) {
+        throw new HoodieMetadataException("Failed to commit rollback deletes at instant " + instantTime, e);
+      }
+    });
+
+    partitionToAppendedFiles.forEach((partition, appendedFileMap) -> {
+      fileChangeCount[1] += appendedFileMap.size();
+
+      // Validate that no appended file has been deleted
+      ValidationUtils.checkState(
+          !appendedFileMap.keySet().removeAll(partitionToDeletedFiles.getOrDefault(partition, Collections.emptyList())),
+            "Rollback file cannot both be appended and deleted");
+
+      // New files added to a partition
+      HoodieRecord record = HoodieMetadataPayload.createPartitionFilesRecord(partition, Option.of(appendedFileMap),
+          Option.empty());
+      records.add(record);
+    });
+
+    LOG.info("Updating at " + instantTime + " from " + operation + ". #partitions_updated=" + records.size()
+        + ", #files_deleted=" + fileChangeCount[0] + ", #files_appended=" + fileChangeCount[1]);
+    commit(prepRecords(records, METADATA_PARTITION_NAME), instantTime, true);
+  }
+
+  /**
+   * Commit the {@code HoodieRecord}s to Metadata Table as a new delta-commit.
+   *
+   * @param recordRDD The records to commit
+   * @param instantTime The timestamp of instant to create
+   * @param performUpdate If True, records are upserted. If False, records are inserted
+   */
+  private synchronized void commit(JavaRDD<HoodieRecord> recordRDD, String instantTime, boolean performUpdate) {
+    ValidationUtils.checkState(!readOnly, "Metadata table cannot be committed in readonly mode");
+    //ValidationUtils.checkArgument(!recordRDD.isEmpty());
+
+    // Close all readers
+    if (basefileReader != null) {
+      basefileReader.close();
+      basefileReader = null;
+    }
+    logRecordScanner = null;
+
+    try (HoodieWriteClient writeClient = new HoodieWriteClient(jsc, config, true)) {
+      writeClient.startCommitWithTime(instantTime);
+      List<WriteStatus> statuses;
+      if (performUpdate) {
+        statuses = writeClient.upsertPreppedRecords(recordRDD, instantTime).collect();
+      } else {
+        statuses = writeClient.insertPreppedRecords(recordRDD, instantTime).collect();
+      }
+      statuses.forEach(writeStatus -> {
+        if (writeStatus.hasErrors()) {
+          throw new HoodieMetadataException("Failed to commit metadata table records at instant " + instantTime);
+        }
+      });
+    }
+  }
+
+  /**
+   * Tag each record with the location.
+   *
+   * Since we only read the latest base file in a partition, we tag the records with the instant time of the latest
+   * base file.
+   */
+  private JavaRDD<HoodieRecord> prepRecords(List<HoodieRecord> records, String partitionName) {
+    HoodieTable table = HoodieTable.create(metaClient, config, hadoopConf);
+    SliceView fsView = table.getSliceView();
+    List<HoodieBaseFile> baseFiles = fsView.getLatestFileSlices(partitionName)
+        .map(s -> s.getBaseFile())
+        .filter(b -> b.isPresent())
+        .map(b -> b.get())
+        .collect(Collectors.toList());
+
+    // All the metadata fits within a single base file
+    if (partitionName.equals(METADATA_PARTITION_NAME)) {
+      if (baseFiles.size() > 1) {
+        throw new HoodieMetadataException("Multiple base files found in metadata partition");
+      }
+    }
+
+    String fileId;
+    String instantTime;
+    if (!baseFiles.isEmpty()) {
+      fileId = baseFiles.get(0).getFileId();
+      instantTime = baseFiles.get(0).getCommitTime();
+    } else {
+      // If there is a log file then we can assume that it has the data
+      List<HoodieLogFile> logFiles = fsView.getLatestFileSlices(HoodieMetadataImpl.METADATA_PARTITION_NAME)
+          .map(s -> s.getLatestLogFile())
+          .filter(b -> b.isPresent())
+          .map(b -> b.get())
+          .collect(Collectors.toList());
+      if (logFiles.isEmpty()) {
+        // No base and log files. All are new inserts
+        return jsc.parallelize(records);
+      }
+
+      fileId = logFiles.get(0).getFileId();
+      instantTime = logFiles.get(0).getBaseCommitTime();
+    }
+
+    return jsc.parallelize(records).map(r -> r.setCurrentLocation(new HoodieRecordLocation(instantTime, fileId)));
+  }
+
+  /**
+   * Returns a list of all partitions.
+   */
+  List<String> getAllPartitionPaths() throws IOException {
+    final Timer.Context context = this.metrics.getMetadataCtx(LOOKUP_PARTITIONS_STR);

Review comment:
       metrics will be null for readonly case. It needs to be checked for null 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497684833



##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+
+    StringBuffer out = new StringBuffer("\n");
+    if (!HoodieMetadata.exists(HoodieCLI.basePath)) {
+      out.append("=== Metadata Table not initilized. Using file listing to get list of partitions. ===\n\n");
+    }
+
+    long t1 = System.currentTimeMillis();
+    List<String> partitions = HoodieMetadata.getAllPartitionPaths(HoodieCLI.fs, HoodieCLI.basePath, false);
+    long t2 = System.currentTimeMillis();
+
+    int[] count = {0};
+    partitions.stream().sorted((p1, p2) -> p2.compareTo(p1)).forEach(p -> {
+      out.append(p);
+      if (++count[0] % 15 == 0) {
+        out.append("\n");
+      } else {
+        out.append(", ");
+      }
+    });
+
+    out.append(String.format("\n\n=== List of partitions retrieved in %.2fsec ===", (t2 - t1) / 1000.0));
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-files", help = "Print a list of all files in a partition from the metadata")
+  public String listFiles(
+      @CliOption(key = {"partition"}, help = "Name of the partition to list files", mandatory = true)
+      final String partition) throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();

Review comment:
       > @prashantwason : Also Have you been able to run metadata list commands successfully ? Did you encounter this issue : Caused by: java.lang.ClassNotFoundException: org.apache.hudi.org.apache.hadoop.hbase.KeyValue$KeyComparator ?
   
   Everyone worked for me. I did a complete test before pushing the CLI code. I am checking it again. This seems like a library version issue to me. I am debugging it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [x] 1. Support for rollbacks in MOR Table
   - [x] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [x] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [x] 7. Unit test for rollback of partial commits
   - [x] 8. Schema evolution strategy for metadata table
   - [x] 9. Unit test for marker based rollback
   - [x] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Fix the case when the table is non-partitioned
   - [ ] 13. Test for Async cases
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491331025



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataIndex.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordLocation;
+import org.apache.hudi.common.table.view.TableFileSystemView.SliceView;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIndexException;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.index.HoodieIndexUtils;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import scala.Tuple2;
+
+/**
+ * An Index which is specific to the Hoodie Metadata Table.
+ *
+ * Hoodie Metadata Table saves information in specific named partitions. So the partition and the base file location
+ * during indexing can be determined by looking at the record itself. This speeds up the indexing during updates.
+ */
+public class HoodieMetadataIndex<T extends HoodieMetadataPayload> extends HoodieIndex<T> {
+
+  public HoodieMetadataIndex(HoodieWriteConfig config) {
+    super(config);
+  }
+
+  /**
+   * Returns an RDD mapping each HoodieKey with a partitionPath/fileID which contains it. Option.Empty if the key is not
+   * found.
+   *
+   * @param hoodieKeys  keys to lookup
+   * @param jsc         spark context
+   * @param hoodieTable hoodie table object
+   */
+  @Override
+  public JavaPairRDD<HoodieKey, Option<Pair<String, String>>> fetchRecordLocation(JavaRDD<HoodieKey> hoodieKeys,
+                                                                                  JavaSparkContext jsc,
+                                                                                  HoodieTable<T> hoodieTable) {
+    JavaPairRDD<String, String> partitionRecordKeyPairRDD =
+        hoodieKeys.mapToPair(key -> new Tuple2<>(key.getPartitionPath(), key.getRecordKey()));
+
+    // Lookup location of all the hoodie keys
+    JavaPairRDD<HoodieKey, HoodieRecordLocation> recordKeyLocationRDD = lookupIndex(hoodieKeys, jsc, hoodieTable);
+    JavaPairRDD<HoodieKey, String> keyHoodieKeyPairRDD = hoodieKeys.mapToPair(key -> new Tuple2<>(key, null));
+
+    return keyHoodieKeyPairRDD.leftOuterJoin(recordKeyLocationRDD).mapToPair(keyLoc -> {
+      Option<Pair<String, String>> partitionPathFileidPair;
+      if (keyLoc._2._2.isPresent()) {
+        partitionPathFileidPair = Option.of(Pair.of(keyLoc._1().getPartitionPath(), keyLoc._2._2.get().getFileId()));
+      } else {
+        partitionPathFileidPair = Option.empty();
+      }
+      return new Tuple2<>(keyLoc._1, partitionPathFileidPair);
+    });
+  }
+
+  /**
+   * Tag each record with its current location (partition / fileId).
+   *
+   * @param recordRDD   records to tag
+   * @param jsc         spark context
+   * @param hoodieTable hoodie table object
+   */
+  @Override
+  public JavaRDD<HoodieRecord<T>> tagLocation(JavaRDD<HoodieRecord<T>> recordRDD, JavaSparkContext jsc,
+                                              HoodieTable<T> hoodieTable) throws HoodieIndexException {
+    JavaPairRDD<HoodieKey, HoodieRecord> keyRecordPairRDD = recordRDD.mapToPair(r -> new Tuple2<>(r.getKey(), r));
+    JavaPairRDD<HoodieKey, HoodieRecordLocation> keyLocationPairRDD =
+        lookupIndex(keyRecordPairRDD.keys(), jsc, hoodieTable);
+
+    return keyRecordPairRDD.leftOuterJoin(keyLocationPairRDD).values().map(v1 -> {
+      return HoodieIndexUtils.getTaggedRecord(v1._1, Option.ofNullable(v1._2.orNull()));
+    });
+  }
+
+  /**
+   * Update the index (if required) from the results of the writes into the hoodie table.
+   *
+   * @param writeStatusRDD  status of the writes
+   * @param jsc             spark context
+   * @param hoodieTable     hoodie table object
+   */
+  @Override
+  public JavaRDD<WriteStatus> updateLocation(JavaRDD<WriteStatus> writeStatusRDD, JavaSparkContext jsc,
+                                             HoodieTable<T> hoodieTable)
+      throws HoodieIndexException {
+    // No update required for metadata index
+    return writeStatusRDD;
+  }
+
+  /**
+   * Lookup the location for each hoodie key and return the pair<key, location> for all record keys already
+   * present and drop the record keys if not present.
+   */
+  private JavaPairRDD<HoodieKey, HoodieRecordLocation> lookupIndex(JavaRDD<HoodieKey> hoodieKeys,
+                                                                   JavaSparkContext jsc,
+                                                                   HoodieTable<T> hoodieTable) {
+    SliceView fsView = hoodieTable.getSliceView();
+    List<HoodieBaseFile> baseFiles = fsView.getLatestFileSlices(HoodieMetadataImpl.METADATA_PARTITION_NAME)

Review comment:
       Yes. Since we dont have other metadata table partitions, I have kept it simple.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [x] 1. Support for rollbacks in MOR Table
   - [ ] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [x] 7. Unit test for rollback of partial commits
   - [ ] 8. Schema evolution strategy for metadata table
   - [x] 9. Unit test for marker based rollback
   - [ ] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason edited a comment on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason edited a comment on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-686688968


   Remaining work items:
   
   - [x] 1. Support for rollbacks in MOR Table
   - [ ] 2. Rollback of metadata if commit eventually fails on dataset 
   - [x] 3. HUDI-CLI extensions for metadata debugging
   - [ ] 4. Ensure partial rollbacks do not use metadata table as it does not contain partial info 
   - [ ] 5. Fix initialization when Async jobs are scheduled - these jobs have older timestamp than INIT timestamp on metadata table
   - [ ] 6. Check if MergedBlockReader will neglect log blocks based on uncommitted commits.
   - [ ] 7. Unit test for rollback of partial commits
   - [ ] 8. Schema evolution strategy for metadata table
   - [x] 9. Unit test for marker based rollback
   - [ ] 10. Can all compaction strategies work off of metadata table itself? Does it have all the data
   - [ ] 11. Async Clean and Async Compaction - how will they work with metadata table updates - check multi writer
   - [ ] 12. Query-side use of metadata table
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r491341159



##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -253,6 +254,15 @@ public SyncableFileSystemView getHoodieView() {
     return getViewManager().getFileSystemView(metaClient);
   }
 
+  private SyncableFileSystemView getFileSystemViewInternal(HoodieTimeline timeline) {
+    if (config.useFileListingMetadata()) {
+      FileSystemViewStorageConfig viewConfig = config.getViewStorageConfig();
+      return new HoodieMetadataFileSystemView(metaClient, timeline, viewConfig.isIncrementalTimelineSyncEnabled());
+    } else {
+      return getViewManager().getFileSystemView(metaClient);

Review comment:
       Not currently as entire metadata code is within hudi-client and viewManager is within hudi-common.
   
   I will have to eventually restructure into metadata-reader and metadata-writer and I will make this change then.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

bvaradar commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r497149559



##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();

Review comment:
       > > I instrumented the code to track writeConfig.useFileListingMetadata. It looks like it is being disabled for a dataset that has the metadata dataset
   > > 20/09/26 02:58:11 ERROR HoodieMetadata: writeConfig.useFileListingMetadata() =true
   > > 20/09/26 02:58:14 ERROR HoodieMetadata: writeConfig.useFileListingMetadata() =false
   > > The above line is coming from HoodieMetadata.init() method. I am unable to test using spark.write() as it seems to fallback to listing
   > 
   > I think the second log (writeConfig.useFileListingMetadata() =false) is coming from the Hoodie Metadata Table's internal writeConfig. Since the HUDI Metadata Table is also a HUDI Table, it also has a HoodieWriteConfig wherein the useFileListingMetadata is disabled.
   > 
   > Fallback to listing is by design to prevent issues during adoption.
   
   Got it, So, the spark.write.xxxx() works ? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-692389993


   @prashantwason please let @umehrot2 know if we can start testing this. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-696429882






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r501383798



##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/MetadataCommand.java
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.cli.commands;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.cli.HoodieCLI;
+import org.apache.hudi.cli.utils.SparkUtil;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metadata.HoodieMetadata;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.springframework.shell.core.CommandMarker;
+import org.springframework.shell.core.annotation.CliCommand;
+import org.springframework.shell.core.annotation.CliOption;
+import org.springframework.stereotype.Component;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * CLI commands to operate on the Metadata Table.
+ */
+@Component
+public class MetadataCommand implements CommandMarker {
+  private JavaSparkContext jsc;
+
+  @CliCommand(value = "metadata set", help = "Set options for Metadata Table")
+  public String set(@CliOption(key = {"metadataDir"},
+      help = "Directory to read/write metadata table (can be different from dataset)", unspecifiedDefaultValue = "")
+      final String metadataDir) {
+    if (!metadataDir.isEmpty()) {
+      HoodieMetadata.setMetadataBaseDirectory(metadataDir);
+    }
+
+    return String.format("Ok");
+  }
+
+  @CliCommand(value = "metadata create", help = "Create the Metadata Table if it does not exist")
+  public String create() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") not empty.");
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist yet
+      HoodieCLI.fs.mkdirs(metadataPath);
+    }
+
+    long t1 = System.currentTimeMillis();
+    HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+        .withUseFileListingMetadata(true).build();
+    initJavaSparkContext();
+    HoodieMetadata.init(jsc, writeConfig);
+    long t2 = System.currentTimeMillis();
+
+    return String.format("Created Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata delete", help = "Remove the Metadata Table")
+  public String delete() throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+      if (statuses.length > 0) {
+        HoodieCLI.fs.delete(metadataPath, true);
+      }
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    return String.format("Removed Metdata Table from %s", metadataPath);
+  }
+
+  @CliCommand(value = "metadata init", help = "Update the metadata table from commits since the creation")
+  public String init(@CliOption(key = {"readonly"}, unspecifiedDefaultValue = "false",
+      help = "Open in read-only mode") final boolean readOnly) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Path metadataPath = new Path(HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath));
+    try {
+      FileStatus[] statuses = HoodieCLI.fs.listStatus(metadataPath);
+    } catch (FileNotFoundException e) {
+      // Metadata directory does not exist
+      throw new RuntimeException("Metadata directory (" + metadataPath.toString() + ") does not exist.");
+    }
+
+    HoodieMetadata.remove(HoodieCLI.basePath);
+
+    long t1 = System.currentTimeMillis();
+    if (readOnly) {
+      HoodieMetadata.init(HoodieCLI.conf, HoodieCLI.basePath);
+    } else {
+      HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+          .withUseFileListingMetadata(true).build();
+      initJavaSparkContext();
+      HoodieMetadata.init(jsc, writeConfig);
+    }
+    long t2 = System.currentTimeMillis();
+
+    String action = readOnly ? "Opened" : "Initialized";
+    return String.format(action + " Metadata Table in %s (duration=%.2fsec)", metadataPath, (t2 - t1) / 1000.0);
+  }
+
+  @CliCommand(value = "metadata stats", help = "Print stats about the metadata")
+  public String stats() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+    Map<String, String> stats = HoodieMetadata.getStats(HoodieCLI.basePath, true);
+
+    StringBuffer out = new StringBuffer("\n");
+    out.append(String.format("Base path: %s\n", HoodieMetadata.getMetadataTableBasePath(HoodieCLI.basePath)));
+    for (Map.Entry<String, String> entry : stats.entrySet()) {
+      out.append(String.format("%s: %s\n", entry.getKey(), entry.getValue()));
+    }
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-partitions", help = "Print a list of all partitions from the metadata")
+  public String listPartitions() throws IOException {
+    HoodieTableMetaClient metaClient = HoodieCLI.getTableMetaClient();
+
+    StringBuffer out = new StringBuffer("\n");
+    if (!HoodieMetadata.exists(HoodieCLI.basePath)) {
+      out.append("=== Metadata Table not initilized. Using file listing to get list of partitions. ===\n\n");
+    }
+
+    long t1 = System.currentTimeMillis();
+    List<String> partitions = HoodieMetadata.getAllPartitionPaths(HoodieCLI.fs, HoodieCLI.basePath, false);
+    long t2 = System.currentTimeMillis();
+
+    int[] count = {0};
+    partitions.stream().sorted((p1, p2) -> p2.compareTo(p1)).forEach(p -> {
+      out.append(p);
+      if (++count[0] % 15 == 0) {
+        out.append("\n");
+      } else {
+        out.append(", ");
+      }
+    });
+
+    out.append(String.format("\n\n=== List of partitions retrieved in %.2fsec ===", (t2 - t1) / 1000.0));
+
+    return out.toString();
+  }
+
+  @CliCommand(value = "metadata list-files", help = "Print a list of all files in a partition from the metadata")
+  public String listFiles(

Review comment:
       Yes. Add a unit test too.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] prashantwason commented on a change in pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

prashantwason commented on a change in pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#discussion_r498587425



##########
File path: hudi-client/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieMetadataException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * This is a payload which saves information about a single entry in the Metadata Table. The type of the entry is
+ * determined by the "type" saved within the record. The following types of entries are saved:
+ *
+ *   1. List of partitions: There is a single such record
+ *         key="__all_partitions__"
+ *         filenameToSizeMap={"2020/01/01": 0, "2020/01/02": 0, ...}
+ *
+ *   2. List of files in a Partition: There is one such record for each partition
+ *         key=Partition name
+ *         filenameToSizeMap={"file1.parquet": 12345, "file2.parquet": 56789, "file1.log": 9876,
+ *                            "file0.parquet": -1, ...}
+ *
+ *      For deleted files, -1 is used as the size.
+ *
+ *  During compaction on the table, the deletions are merged with additions and hence pruned.
+ */
+public class HoodieMetadataPayload implements HoodieRecordPayload<HoodieMetadataPayload> {
+  private static final Logger LOG = LogManager.getLogger(HoodieMetadataPayload.class);
+
+  // Represents the size stored for a deleted file
+  private static final long DELETED_FILE_SIZE = -1;
+
+  // Key and type for the metadata record
+  private final String metadataKey;
+  private final PayloadType type;
+
+  // Filenames which are part of this record
+  // key=filename, value=file size (or DELETED_FILE_SIZE to represent a deleted file)
+  private final Map<String, Long> filenameMap = new HashMap<>();
+
+  // Type of the metadata record
+  public enum PayloadType {
+    PARTITION_LIST(1),        // list of partitions
+    PARTITION_FILES(2);       // list of files in a partition
+
+    private final int value;

Review comment:
       I have changed the schema. PTAL.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #2064: WIP - [HUDI-842] Implementation of HUDI RFC-15.

[GitBox <gi...@apache.org>]

vinothchandar commented on pull request #2064:
URL: https://github.com/apache/hudi/pull/2064#issuecomment-696429882


   cc @bvaradar @n3nash as well
   
   @prashantwason Here is a corner case with syncing completed compaction from data timeline to metadata timeline. Consider the following sequence of events 
   
   t0: writer schedules compaction at time instant `c`
   t1: Compactor starts processing `c`'s plan
   t2: compaction finishes with `c.commit` published on the data timeline (not yet synced to metadata timeline) 
   t3: Next round of writing, writer opens metadata table, which adds the base file produced in c.commit to metadata table.
   
   Any queries running between t2 and t3, cannot rely on metadata since the new base file will not be present in metadata table. The timeline will indicate that the compaction completed, and the latest file slice will be computed  as simply the logs written to the file groups since compaction. This will lead to incorrect results. 
   
   If we consider just writer alone, we may be okay since we first sync the metadata table before we do anything for the delta commit at t3. But in general for queries, we should advise enabling metadata table based listings only, after all writers/cleaner/compactor have been enabled to use metadata and been successfully using it to publish new/deleted files directly to the metadata table. In short, queries cannot rely on metadata table, with the syncing mechanism as the main thing that keeps data and metadata timelines together. 
   
   
   
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org