You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2023/12/06 20:05:03 UTC

[PR] [HUDI-7176] Add file group reader test framework [hudi]

linliu-code opened a new pull request, #10263:
URL: https://github.com/apache/hudi/pull/10263

   Our goal is to add a framework for unit tests at the level of file group reading. That is,
   we don't need to involve any engine specific logic during the read, such that we could
   
   1. Simplify the test process by providing unified and simple APIs,
   2. Impose more granular read process for testing, and
   3. Improve the test performance since no engine specific logic are involved.
   
   We will add the framework step by step.
   
   ### Change Logs
   
   Steps:
   1. Add a builder class to construct file group reader.
   
   ### Impact
   
   1. Simplify the process to construct file group reader level tests;
   5. Improve the test performance by removing engine specific logic.
   
   ### Risk level (write none, low medium or high below)
   
   None.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1424492464


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {

Review Comment:
   I plan to have another PR to further extract the framework, I can integrate it to the `HoodieTestTable` then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1847654055

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1424490035


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieTestReaderContext.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecordMerger;
+import org.apache.hudi.common.model.HoodieEmptyRecord;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieOperation;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+
+import static org.apache.hudi.common.model.HoodieRecordMerger.DEFAULT_MERGER_STRATEGY_UUID;
+import static org.apache.hudi.common.testutils.reader.HoodieFileSliceTestUtils.ROW_KEY;
+
+public class HoodieTestReaderContext extends HoodieReaderContext<IndexedRecord> {
+  @Override
+  public FileSystem getFs(String path, Configuration conf) {
+    return FSUtils.getFs(path, conf);
+  }
+
+  @Override
+  public ClosableIterator<IndexedRecord> getFileRecordIterator(
+      Path filePath,
+      long start,
+      long length,
+      Schema dataSchema,
+      Schema requiredSchema,
+      Configuration conf
+  ) throws IOException {
+    HoodieAvroParquetReader reader = new HoodieAvroParquetReader(conf, filePath);
+    return reader.getIndexedRecordIterator(dataSchema, requiredSchema);
+  }
+
+  @Override
+  public IndexedRecord convertAvroRecord(IndexedRecord record) {
+    return record;
+  }
+
+  @Override
+  public HoodieRecordMerger getRecordMerger(String mergerStrategy) {
+    switch (mergerStrategy) {
+      case DEFAULT_MERGER_STRATEGY_UUID:
+        return new HoodieAvroRecordMerger();
+      default:
+        throw new HoodieException(
+            "The merger strategy UUID is not supported: " + mergerStrategy);
+    }
+  }
+
+  @Override
+  public Object getValue(IndexedRecord record, Schema schema, String fieldName) {
+    return getFieldValueFromIndexedRecord(record, schema, fieldName);
+  }
+
+  @Override
+  public String getRecordKey(IndexedRecord record, Schema schema) {
+    return getFieldValueFromIndexedRecord(record, schema, ROW_KEY).toString();
+  }
+
+  @Override
+  public Comparable getOrderingValue(
+      Option<IndexedRecord> recordOpt,
+      Map<String, Object> metadataMap,
+      Schema schema,
+      TypedProperties props
+  ) {
+    if (metadataMap.containsKey(INTERNAL_META_ORDERING_FIELD)) {
+      return (Comparable) metadataMap.get(INTERNAL_META_ORDERING_FIELD);
+    }
+
+    if (!recordOpt.isPresent()) {
+      return 0;
+    }
+
+    String orderingFieldName = ConfigUtils.getOrderingField(props);
+    Object value = getFieldValueFromIndexedRecord(recordOpt.get(), schema, orderingFieldName);
+    return value != null ? (Comparable) value : 0;
+  }
+
+  @Override
+  public HoodieRecord<IndexedRecord> constructHoodieRecord(
+      Option<IndexedRecord> recordOpt,
+      Map<String, Object> metadataMap
+  ) {
+    if (!recordOpt.isPresent()) {
+      HoodieKey key = new HoodieKey((String) metadataMap.get(INTERNAL_META_RECORD_KEY),
+          (String) metadataMap.get(INTERNAL_META_PARTITION_PATH));
+      return new HoodieEmptyRecord<>(
+          key,
+          HoodieOperation.DELETE,
+          (Comparable<?>) metadataMap.get(INTERNAL_META_ORDERING_FIELD),
+          HoodieRecord.HoodieRecordType.AVRO);
+    }
+    return new HoodieAvroIndexedRecord(recordOpt.get());
+  }
+
+  @Override
+  public IndexedRecord seal(IndexedRecord record) {
+    Schema schema = record.getSchema();
+    GenericRecordBuilder builder = new GenericRecordBuilder(schema);
+    for (Schema.Field field : schema.getFields()) {
+      builder.set(field, record.get(field.pos()));
+    }
+    return builder.build();
+  }
+
+  @Override
+  public ClosableIterator<IndexedRecord> mergeBootstrapReaders(ClosableIterator<IndexedRecord> skeletonFileIterator, ClosableIterator<IndexedRecord> dataFileIterator) {
+    return null;
+  }
+
+  @Override
+  public UnaryOperator<IndexedRecord> projectRecord(Schema from, Schema to) {
+    return null;
+  }

Review Comment:
   Yeah, this is introduced in Jon recently. I have implemented in another PR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1420893953


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/DataGenerationPlan.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import java.util.List;
+
+/**
+ * The blueprint of records that will be generated
+ * by the data generator.
+ */
+public class DataGenerationPlan {
+  // The values for "_row_key" field.
+  private final List<String> recordKeys;
+  // The partition path for all records.
+  private final String partitionPath;
+  // The ordering field.
+  private final long timestamp;
+  // The operation type of the record.
+  private final OperationType operationType;
+  private final String instantTime;
+
+  public enum OperationType {
+    INSERT,
+    UPDATE,
+    DELETE
+  }
+
+  public DataGenerationPlan(List<String> recordKeys,
+                            String partitionPath,

Review Comment:
   We seem to be limited to only a single partition path? What if we want multiple partitions in a write?



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";
+  public static final String PARQUET = ".parquet";
+  public static final String ROW_KEY = "_row_key";
+  public static final String PARTITION_PATH = "partition_path";
+  public static final String RIDER = "rider";
+  public static final String DRIVER = "driver";
+  public static final String TIMESTAMP = "timestamp";
+  public static final HoodieTestDataGenerator DATA_GEN =
+      new HoodieTestDataGenerator(0xDEED);
+
+  // We use a number to represent a record key, and a (start, end) range
+  // to represent a set of record keys between start <= k <= end.
+  public static class KeyRange {
+    public int start;
+    public int end;
+
+    public KeyRange(int start, int end) {
+      this.start = start;
+      this.end = end;
+    }
+  }
+
+  private static Path generateBaseFilePath(
+      String basePath,
+      String fileId,
+      String instantTime
+  ) {
+    return new Path(
+        basePath + FORWARD_SLASH
+        + baseFileName(instantTime, fileId, PARQUET));
+  }
+
+  private static Path generateLogFilePath(
+      String basePath,
+      String fileId,
+      String instantTime,
+      int version) {
+    return new Path(
+        basePath + FORWARD_SLASH + logFileName(
+        instantTime, fileId, version));
+  }
+
+  // Note:
+  // "start < end" means start <= k <= end.
+  // "start == end" means k = start.
+  // "start > end" means no keys.
+  private static List<String> generateKeys(KeyRange range) {
+    List<String> keys = new ArrayList<>();
+    if (range.start == range.end) {
+      keys.add(String.valueOf(range.start));
+    } else {
+      keys = IntStream
+          .rangeClosed(range.start, range.end)
+          .boxed()
+          .map(String::valueOf).collect(Collectors.toList());
+    }
+    return keys;
+  }
+
+  private static List<IndexedRecord> generateRecords(DataGenerationPlan plan) {
+    List<IndexedRecord> records = new ArrayList<>();
+    List<String> keys = plan.getRecordKeys();
+    for (String key : keys) {
+      records.add(DATA_GEN.generateGenericRecord(

Review Comment:
   Can we add this logic into the datagenerator itself?



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";

Review Comment:
   Not a fan of this.  What is your reasoning for needing this?



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";
+  public static final String PARQUET = ".parquet";
+  public static final String ROW_KEY = "_row_key";
+  public static final String PARTITION_PATH = "partition_path";
+  public static final String RIDER = "rider";
+  public static final String DRIVER = "driver";
+  public static final String TIMESTAMP = "timestamp";
+  public static final HoodieTestDataGenerator DATA_GEN =
+      new HoodieTestDataGenerator(0xDEED);
+
+  // We use a number to represent a record key, and a (start, end) range
+  // to represent a set of record keys between start <= k <= end.
+  public static class KeyRange {
+    public int start;
+    public int end;
+
+    public KeyRange(int start, int end) {
+      this.start = start;
+      this.end = end;
+    }
+  }
+
+  private static Path generateBaseFilePath(
+      String basePath,
+      String fileId,
+      String instantTime
+  ) {
+    return new Path(
+        basePath + FORWARD_SLASH
+        + baseFileName(instantTime, fileId, PARQUET));
+  }
+
+  private static Path generateLogFilePath(
+      String basePath,
+      String fileId,
+      String instantTime,
+      int version) {
+    return new Path(
+        basePath + FORWARD_SLASH + logFileName(
+        instantTime, fileId, version));
+  }
+
+  // Note:
+  // "start < end" means start <= k <= end.
+  // "start == end" means k = start.
+  // "start > end" means no keys.
+  private static List<String> generateKeys(KeyRange range) {
+    List<String> keys = new ArrayList<>();
+    if (range.start == range.end) {
+      keys.add(String.valueOf(range.start));
+    } else {
+      keys = IntStream
+          .rangeClosed(range.start, range.end)
+          .boxed()
+          .map(String::valueOf).collect(Collectors.toList());
+    }
+    return keys;
+  }
+
+  private static List<IndexedRecord> generateRecords(DataGenerationPlan plan) {
+    List<IndexedRecord> records = new ArrayList<>();
+    List<String> keys = plan.getRecordKeys();
+    for (String key : keys) {
+      records.add(DATA_GEN.generateGenericRecord(
+          key,
+          plan.getPartitionPath(),
+          RIDER + PERIOD + UUID.randomUUID().toString(),
+          DRIVER + PERIOD + UUID.randomUUID().toString(),
+          plan.getTimestamp(),
+          plan.getOperationType() == DELETE,
+          false
+      ));
+    }
+    return records;
+  }
+
+  private static HoodieDataBlock getDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path logFilePath
+  ) {
+    return createDataBlock(
+        dataBlockType,
+        records.stream().map(HoodieAvroIndexedRecord::new)
+            .collect(Collectors.toList()),
+        header,
+        logFilePath);
+  }
+
+  private static HoodieDataBlock createDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<HoodieRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path pathForReader
+  ) {
+    switch (dataBlockType) {
+      case CDC_DATA_BLOCK:
+        return new HoodieCDCDataBlock(
+            records,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case AVRO_DATA_BLOCK:
+        return new HoodieAvroDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case HFILE_DATA_BLOCK:
+        return new HoodieHFileDataBlock(
+            records,
+            header,
+            Compression.Algorithm.GZ,
+            pathForReader);
+      case PARQUET_DATA_BLOCK:
+        return new HoodieParquetDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD,
+            CompressionCodecName.GZIP,
+            0.1,
+            true);
+      default:
+        throw new RuntimeException(
+            "Unknown data block type " + dataBlockType);
+    }
+  }
+
+  public static HoodieDeleteBlock getDeleteBlock(
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Schema schema,
+      Properties props
+  ) {
+    List<HoodieRecord> hoodieRecords = records.stream()
+        .map(r -> {
+          String rowKey = (String) r.get(r.getSchema().getField(ROW_KEY).pos());
+          String partitionPath = (String) r.get(r.getSchema().getField(PARTITION_PATH).pos());
+          return new HoodieAvroIndexedRecord(new HoodieKey(rowKey, partitionPath), r);
+        })
+        .collect(Collectors.toList());
+    return new HoodieDeleteBlock(
+        hoodieRecords.stream().map(
+            r -> Pair.of(DeleteRecord.create(
+                r.getKey(), r.getOrderingValue(schema, props)), -1L))
+            .collect(Collectors.toList()),
+        false,
+        header
+    );
+  }
+
+  public static HoodieBaseFile createBaseFile(
+      String baseFilePath,
+      List<IndexedRecord> records,
+      Schema schema,
+      String baseInstantTime
+  ) throws IOException {
+    Configuration hadoopConf = new Configuration();
+
+    BloomFilter filter = BloomFilterFactory.createBloomFilter(
+        1000,
+        0.0001,
+        10000,
+        BloomFilterTypeCode.DYNAMIC_V0.name());
+    HoodieAvroWriteSupport<IndexedRecord> writeSupport = new HoodieAvroWriteSupport<>(
+        new AvroSchemaConverter().convert(schema),
+        schema,
+        Option.of(filter),
+        new Properties());
+    HoodieParquetConfig<HoodieAvroWriteSupport> parquetConfig = new HoodieParquetConfig(
+        writeSupport,
+        CompressionCodecName.GZIP,
+        ParquetWriter.DEFAULT_BLOCK_SIZE,
+        ParquetWriter.DEFAULT_PAGE_SIZE,
+        1024 * 1024 * 1024,
+        hadoopConf,
+        0.1,
+        true);
+

Review Comment:
   Can you explain what we are doing here? Lots of hard coded numbers



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";
+  public static final String PARQUET = ".parquet";
+  public static final String ROW_KEY = "_row_key";
+  public static final String PARTITION_PATH = "partition_path";
+  public static final String RIDER = "rider";
+  public static final String DRIVER = "driver";
+  public static final String TIMESTAMP = "timestamp";
+  public static final HoodieTestDataGenerator DATA_GEN =
+      new HoodieTestDataGenerator(0xDEED);
+
+  // We use a number to represent a record key, and a (start, end) range
+  // to represent a set of record keys between start <= k <= end.
+  public static class KeyRange {
+    public int start;
+    public int end;
+
+    public KeyRange(int start, int end) {
+      this.start = start;
+      this.end = end;
+    }
+  }
+
+  private static Path generateBaseFilePath(
+      String basePath,
+      String fileId,
+      String instantTime
+  ) {
+    return new Path(
+        basePath + FORWARD_SLASH
+        + baseFileName(instantTime, fileId, PARQUET));
+  }
+
+  private static Path generateLogFilePath(
+      String basePath,
+      String fileId,
+      String instantTime,
+      int version) {
+    return new Path(
+        basePath + FORWARD_SLASH + logFileName(
+        instantTime, fileId, version));
+  }
+
+  // Note:
+  // "start < end" means start <= k <= end.
+  // "start == end" means k = start.
+  // "start > end" means no keys.
+  private static List<String> generateKeys(KeyRange range) {
+    List<String> keys = new ArrayList<>();
+    if (range.start == range.end) {
+      keys.add(String.valueOf(range.start));
+    } else {
+      keys = IntStream
+          .rangeClosed(range.start, range.end)
+          .boxed()
+          .map(String::valueOf).collect(Collectors.toList());
+    }
+    return keys;
+  }
+
+  private static List<IndexedRecord> generateRecords(DataGenerationPlan plan) {
+    List<IndexedRecord> records = new ArrayList<>();
+    List<String> keys = plan.getRecordKeys();
+    for (String key : keys) {
+      records.add(DATA_GEN.generateGenericRecord(
+          key,
+          plan.getPartitionPath(),
+          RIDER + PERIOD + UUID.randomUUID().toString(),
+          DRIVER + PERIOD + UUID.randomUUID().toString(),
+          plan.getTimestamp(),
+          plan.getOperationType() == DELETE,
+          false
+      ));
+    }
+    return records;
+  }
+
+  private static HoodieDataBlock getDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path logFilePath
+  ) {
+    return createDataBlock(
+        dataBlockType,
+        records.stream().map(HoodieAvroIndexedRecord::new)
+            .collect(Collectors.toList()),
+        header,
+        logFilePath);
+  }
+
+  private static HoodieDataBlock createDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<HoodieRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path pathForReader
+  ) {
+    switch (dataBlockType) {
+      case CDC_DATA_BLOCK:
+        return new HoodieCDCDataBlock(
+            records,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case AVRO_DATA_BLOCK:
+        return new HoodieAvroDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case HFILE_DATA_BLOCK:
+        return new HoodieHFileDataBlock(
+            records,
+            header,
+            Compression.Algorithm.GZ,
+            pathForReader);
+      case PARQUET_DATA_BLOCK:
+        return new HoodieParquetDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD,
+            CompressionCodecName.GZIP,
+            0.1,
+            true);
+      default:
+        throw new RuntimeException(
+            "Unknown data block type " + dataBlockType);
+    }
+  }
+
+  public static HoodieDeleteBlock getDeleteBlock(
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Schema schema,
+      Properties props
+  ) {
+    List<HoodieRecord> hoodieRecords = records.stream()
+        .map(r -> {
+          String rowKey = (String) r.get(r.getSchema().getField(ROW_KEY).pos());
+          String partitionPath = (String) r.get(r.getSchema().getField(PARTITION_PATH).pos());
+          return new HoodieAvroIndexedRecord(new HoodieKey(rowKey, partitionPath), r);
+        })
+        .collect(Collectors.toList());
+    return new HoodieDeleteBlock(
+        hoodieRecords.stream().map(
+            r -> Pair.of(DeleteRecord.create(
+                r.getKey(), r.getOrderingValue(schema, props)), -1L))
+            .collect(Collectors.toList()),
+        false,
+        header
+    );
+  }
+
+  public static HoodieBaseFile createBaseFile(
+      String baseFilePath,
+      List<IndexedRecord> records,
+      Schema schema,
+      String baseInstantTime
+  ) throws IOException {
+    Configuration hadoopConf = new Configuration();
+
+    BloomFilter filter = BloomFilterFactory.createBloomFilter(
+        1000,
+        0.0001,
+        10000,
+        BloomFilterTypeCode.DYNAMIC_V0.name());
+    HoodieAvroWriteSupport<IndexedRecord> writeSupport = new HoodieAvroWriteSupport<>(
+        new AvroSchemaConverter().convert(schema),
+        schema,
+        Option.of(filter),
+        new Properties());
+    HoodieParquetConfig<HoodieAvroWriteSupport> parquetConfig = new HoodieParquetConfig(
+        writeSupport,
+        CompressionCodecName.GZIP,
+        ParquetWriter.DEFAULT_BLOCK_SIZE,
+        ParquetWriter.DEFAULT_PAGE_SIZE,
+        1024 * 1024 * 1024,
+        hadoopConf,
+        0.1,
+        true);
+
+    try (HoodieAvroParquetWriter writer = new HoodieAvroParquetWriter(
+        new Path(baseFilePath),
+        parquetConfig,
+        baseInstantTime,
+        new LocalTaskContextSupplier(),
+        true)) {
+      for (IndexedRecord record : records) {
+        writer.writeAvro(
+            (String) record.get(schema.getField(ROW_KEY).pos()), record);
+      }
+    }
+    return new HoodieBaseFile(baseFilePath);
+  }
+
+  public static HoodieLogFile createLogFile(
+      FileSystem fileSystem,
+      String logFilePath,
+      List<IndexedRecord> records,
+      Schema schema,
+      String fileId,
+      String logInstantTime,
+      int version,
+      HoodieLogBlock.HoodieLogBlockType blockType
+  ) throws InterruptedException, IOException {
+    try (HoodieLogFormat.Writer writer =
+             HoodieLogFormat.newWriterBuilder()
+                 .onParentPath(new Path(logFilePath).getParent())
+                 .withFileExtension(HoodieLogFile.DELTA_EXTENSION)
+                 .withFileId(fileId)
+                 .withDeltaCommit(logInstantTime)
+                 .withLogVersion(version)
+                 .withFs(fileSystem).build()) {
+      Map<HoodieLogBlock.HeaderMetadataType, String> header = new HashMap<>();
+      header.put(HoodieLogBlock.HeaderMetadataType.INSTANT_TIME, logInstantTime);
+      header.put(HoodieLogBlock.HeaderMetadataType.SCHEMA, schema.toString());
+      // TODO: use global properties instead.
+      TypedProperties properties = new TypedProperties();
+      properties.setProperty(
+          "hoodie.datasource.write.precombine.field", "timestamp");

Review Comment:
   should use static TIMESTAMP var 



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";
+  public static final String PARQUET = ".parquet";
+  public static final String ROW_KEY = "_row_key";
+  public static final String PARTITION_PATH = "partition_path";
+  public static final String RIDER = "rider";
+  public static final String DRIVER = "driver";
+  public static final String TIMESTAMP = "timestamp";
+  public static final HoodieTestDataGenerator DATA_GEN =
+      new HoodieTestDataGenerator(0xDEED);
+
+  // We use a number to represent a record key, and a (start, end) range
+  // to represent a set of record keys between start <= k <= end.
+  public static class KeyRange {
+    public int start;
+    public int end;
+
+    public KeyRange(int start, int end) {
+      this.start = start;
+      this.end = end;
+    }
+  }
+
+  private static Path generateBaseFilePath(
+      String basePath,
+      String fileId,
+      String instantTime
+  ) {
+    return new Path(
+        basePath + FORWARD_SLASH
+        + baseFileName(instantTime, fileId, PARQUET));
+  }
+
+  private static Path generateLogFilePath(
+      String basePath,
+      String fileId,
+      String instantTime,
+      int version) {
+    return new Path(
+        basePath + FORWARD_SLASH + logFileName(
+        instantTime, fileId, version));
+  }
+
+  // Note:
+  // "start < end" means start <= k <= end.
+  // "start == end" means k = start.
+  // "start > end" means no keys.
+  private static List<String> generateKeys(KeyRange range) {
+    List<String> keys = new ArrayList<>();
+    if (range.start == range.end) {
+      keys.add(String.valueOf(range.start));
+    } else {
+      keys = IntStream
+          .rangeClosed(range.start, range.end)
+          .boxed()
+          .map(String::valueOf).collect(Collectors.toList());
+    }
+    return keys;
+  }
+
+  private static List<IndexedRecord> generateRecords(DataGenerationPlan plan) {
+    List<IndexedRecord> records = new ArrayList<>();
+    List<String> keys = plan.getRecordKeys();
+    for (String key : keys) {
+      records.add(DATA_GEN.generateGenericRecord(
+          key,
+          plan.getPartitionPath(),
+          RIDER + PERIOD + UUID.randomUUID().toString(),
+          DRIVER + PERIOD + UUID.randomUUID().toString(),
+          plan.getTimestamp(),
+          plan.getOperationType() == DELETE,
+          false
+      ));
+    }
+    return records;
+  }
+
+  private static HoodieDataBlock getDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path logFilePath
+  ) {
+    return createDataBlock(
+        dataBlockType,
+        records.stream().map(HoodieAvroIndexedRecord::new)
+            .collect(Collectors.toList()),
+        header,
+        logFilePath);
+  }
+
+  private static HoodieDataBlock createDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<HoodieRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path pathForReader
+  ) {
+    switch (dataBlockType) {
+      case CDC_DATA_BLOCK:
+        return new HoodieCDCDataBlock(
+            records,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case AVRO_DATA_BLOCK:
+        return new HoodieAvroDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case HFILE_DATA_BLOCK:
+        return new HoodieHFileDataBlock(
+            records,
+            header,
+            Compression.Algorithm.GZ,
+            pathForReader);
+      case PARQUET_DATA_BLOCK:
+        return new HoodieParquetDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD,
+            CompressionCodecName.GZIP,
+            0.1,
+            true);
+      default:
+        throw new RuntimeException(
+            "Unknown data block type " + dataBlockType);
+    }
+  }
+
+  public static HoodieDeleteBlock getDeleteBlock(
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Schema schema,
+      Properties props
+  ) {
+    List<HoodieRecord> hoodieRecords = records.stream()
+        .map(r -> {
+          String rowKey = (String) r.get(r.getSchema().getField(ROW_KEY).pos());
+          String partitionPath = (String) r.get(r.getSchema().getField(PARTITION_PATH).pos());
+          return new HoodieAvroIndexedRecord(new HoodieKey(rowKey, partitionPath), r);
+        })
+        .collect(Collectors.toList());
+    return new HoodieDeleteBlock(
+        hoodieRecords.stream().map(
+            r -> Pair.of(DeleteRecord.create(
+                r.getKey(), r.getOrderingValue(schema, props)), -1L))
+            .collect(Collectors.toList()),
+        false,
+        header
+    );
+  }
+
+  public static HoodieBaseFile createBaseFile(
+      String baseFilePath,
+      List<IndexedRecord> records,
+      Schema schema,
+      String baseInstantTime
+  ) throws IOException {
+    Configuration hadoopConf = new Configuration();
+
+    BloomFilter filter = BloomFilterFactory.createBloomFilter(
+        1000,
+        0.0001,
+        10000,
+        BloomFilterTypeCode.DYNAMIC_V0.name());
+    HoodieAvroWriteSupport<IndexedRecord> writeSupport = new HoodieAvroWriteSupport<>(
+        new AvroSchemaConverter().convert(schema),
+        schema,
+        Option.of(filter),
+        new Properties());
+    HoodieParquetConfig<HoodieAvroWriteSupport> parquetConfig = new HoodieParquetConfig(
+        writeSupport,
+        CompressionCodecName.GZIP,
+        ParquetWriter.DEFAULT_BLOCK_SIZE,
+        ParquetWriter.DEFAULT_PAGE_SIZE,
+        1024 * 1024 * 1024,
+        hadoopConf,
+        0.1,
+        true);
+
+    try (HoodieAvroParquetWriter writer = new HoodieAvroParquetWriter(
+        new Path(baseFilePath),
+        parquetConfig,
+        baseInstantTime,
+        new LocalTaskContextSupplier(),
+        true)) {
+      for (IndexedRecord record : records) {
+        writer.writeAvro(
+            (String) record.get(schema.getField(ROW_KEY).pos()), record);
+      }
+    }
+    return new HoodieBaseFile(baseFilePath);
+  }
+
+  public static HoodieLogFile createLogFile(
+      FileSystem fileSystem,
+      String logFilePath,
+      List<IndexedRecord> records,
+      Schema schema,
+      String fileId,
+      String logInstantTime,
+      int version,
+      HoodieLogBlock.HoodieLogBlockType blockType
+  ) throws InterruptedException, IOException {
+    try (HoodieLogFormat.Writer writer =
+             HoodieLogFormat.newWriterBuilder()
+                 .onParentPath(new Path(logFilePath).getParent())
+                 .withFileExtension(HoodieLogFile.DELTA_EXTENSION)
+                 .withFileId(fileId)
+                 .withDeltaCommit(logInstantTime)
+                 .withLogVersion(version)
+                 .withFs(fileSystem).build()) {
+      Map<HoodieLogBlock.HeaderMetadataType, String> header = new HashMap<>();
+      header.put(HoodieLogBlock.HeaderMetadataType.INSTANT_TIME, logInstantTime);
+      header.put(HoodieLogBlock.HeaderMetadataType.SCHEMA, schema.toString());
+      // TODO: use global properties instead.

Review Comment:
   do todo



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/DataGenerationPlan.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import java.util.List;
+
+/**
+ * The blueprint of records that will be generated
+ * by the data generator.
+ */
+public class DataGenerationPlan {
+  // The values for "_row_key" field.
+  private final List<String> recordKeys;

Review Comment:
   Do we want to test complex keygen or other keygenerators?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #10263:
URL: https://github.com/apache/hudi/pull/10263


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1847716699

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413",
       "triggerID" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413) 
   * 9450ab6f8a73add7f5b25b92c73429ce5f3f25ec UNKNOWN
   * d67c76a2f7fa1b448921ad1deb3cf9b36595650a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1851011224

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413",
       "triggerID" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414",
       "triggerID" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "15f28aa59354f27033840dc140f1e71483257fc4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21455",
       "triggerID" : "15f28aa59354f27033840dc140f1e71483257fc4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9450ab6f8a73add7f5b25b92c73429ce5f3f25ec UNKNOWN
   * d67c76a2f7fa1b448921ad1deb3cf9b36595650a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414) 
   * 15f28aa59354f27033840dc140f1e71483257fc4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21455) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1420922815


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/DataGenerationPlan.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import java.util.List;
+
+/**
+ * The blueprint of records that will be generated
+ * by the data generator.
+ */
+public class DataGenerationPlan {
+  // The values for "_row_key" field.
+  private final List<String> recordKeys;
+  // The partition path for all records.
+  private final String partitionPath;
+  // The ordering field.
+  private final long timestamp;
+  // The operation type of the record.
+  private final OperationType operationType;
+  private final String instantTime;
+
+  public enum OperationType {
+    INSERT,
+    UPDATE,
+    DELETE
+  }
+
+  public DataGenerationPlan(List<String> recordKeys,
+                            String partitionPath,

Review Comment:
   Currently we focus on single partition since we are testing at file slice level, we will add that if needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1848001133

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413",
       "triggerID" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414",
       "triggerID" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9450ab6f8a73add7f5b25b92c73429ce5f3f25ec UNKNOWN
   * d67c76a2f7fa1b448921ad1deb3cf9b36595650a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1420927706


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";
+  public static final String PARQUET = ".parquet";
+  public static final String ROW_KEY = "_row_key";
+  public static final String PARTITION_PATH = "partition_path";
+  public static final String RIDER = "rider";
+  public static final String DRIVER = "driver";
+  public static final String TIMESTAMP = "timestamp";
+  public static final HoodieTestDataGenerator DATA_GEN =
+      new HoodieTestDataGenerator(0xDEED);
+
+  // We use a number to represent a record key, and a (start, end) range
+  // to represent a set of record keys between start <= k <= end.
+  public static class KeyRange {
+    public int start;
+    public int end;
+
+    public KeyRange(int start, int end) {
+      this.start = start;
+      this.end = end;
+    }
+  }
+
+  private static Path generateBaseFilePath(
+      String basePath,
+      String fileId,
+      String instantTime
+  ) {
+    return new Path(
+        basePath + FORWARD_SLASH
+        + baseFileName(instantTime, fileId, PARQUET));
+  }
+
+  private static Path generateLogFilePath(
+      String basePath,
+      String fileId,
+      String instantTime,
+      int version) {
+    return new Path(
+        basePath + FORWARD_SLASH + logFileName(
+        instantTime, fileId, version));
+  }
+
+  // Note:
+  // "start < end" means start <= k <= end.
+  // "start == end" means k = start.
+  // "start > end" means no keys.
+  private static List<String> generateKeys(KeyRange range) {
+    List<String> keys = new ArrayList<>();
+    if (range.start == range.end) {
+      keys.add(String.valueOf(range.start));
+    } else {
+      keys = IntStream
+          .rangeClosed(range.start, range.end)
+          .boxed()
+          .map(String::valueOf).collect(Collectors.toList());
+    }
+    return keys;
+  }
+
+  private static List<IndexedRecord> generateRecords(DataGenerationPlan plan) {
+    List<IndexedRecord> records = new ArrayList<>();
+    List<String> keys = plan.getRecordKeys();
+    for (String key : keys) {
+      records.add(DATA_GEN.generateGenericRecord(
+          key,
+          plan.getPartitionPath(),
+          RIDER + PERIOD + UUID.randomUUID().toString(),
+          DRIVER + PERIOD + UUID.randomUUID().toString(),
+          plan.getTimestamp(),
+          plan.getOperationType() == DELETE,
+          false
+      ));
+    }
+    return records;
+  }
+
+  private static HoodieDataBlock getDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path logFilePath
+  ) {
+    return createDataBlock(
+        dataBlockType,
+        records.stream().map(HoodieAvroIndexedRecord::new)
+            .collect(Collectors.toList()),
+        header,
+        logFilePath);
+  }
+
+  private static HoodieDataBlock createDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<HoodieRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path pathForReader
+  ) {
+    switch (dataBlockType) {
+      case CDC_DATA_BLOCK:
+        return new HoodieCDCDataBlock(
+            records,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case AVRO_DATA_BLOCK:
+        return new HoodieAvroDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case HFILE_DATA_BLOCK:
+        return new HoodieHFileDataBlock(
+            records,
+            header,
+            Compression.Algorithm.GZ,
+            pathForReader);
+      case PARQUET_DATA_BLOCK:
+        return new HoodieParquetDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD,
+            CompressionCodecName.GZIP,
+            0.1,
+            true);
+      default:
+        throw new RuntimeException(
+            "Unknown data block type " + dataBlockType);
+    }
+  }
+
+  public static HoodieDeleteBlock getDeleteBlock(
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Schema schema,
+      Properties props
+  ) {
+    List<HoodieRecord> hoodieRecords = records.stream()
+        .map(r -> {
+          String rowKey = (String) r.get(r.getSchema().getField(ROW_KEY).pos());
+          String partitionPath = (String) r.get(r.getSchema().getField(PARTITION_PATH).pos());
+          return new HoodieAvroIndexedRecord(new HoodieKey(rowKey, partitionPath), r);
+        })
+        .collect(Collectors.toList());
+    return new HoodieDeleteBlock(
+        hoodieRecords.stream().map(
+            r -> Pair.of(DeleteRecord.create(
+                r.getKey(), r.getOrderingValue(schema, props)), -1L))
+            .collect(Collectors.toList()),
+        false,
+        header
+    );
+  }
+
+  public static HoodieBaseFile createBaseFile(
+      String baseFilePath,
+      List<IndexedRecord> records,
+      Schema schema,
+      String baseInstantTime
+  ) throws IOException {
+    Configuration hadoopConf = new Configuration();
+
+    BloomFilter filter = BloomFilterFactory.createBloomFilter(
+        1000,
+        0.0001,
+        10000,
+        BloomFilterTypeCode.DYNAMIC_V0.name());
+    HoodieAvroWriteSupport<IndexedRecord> writeSupport = new HoodieAvroWriteSupport<>(
+        new AvroSchemaConverter().convert(schema),
+        schema,
+        Option.of(filter),
+        new Properties());
+    HoodieParquetConfig<HoodieAvroWriteSupport> parquetConfig = new HoodieParquetConfig(
+        writeSupport,
+        CompressionCodecName.GZIP,
+        ParquetWriter.DEFAULT_BLOCK_SIZE,
+        ParquetWriter.DEFAULT_PAGE_SIZE,
+        1024 * 1024 * 1024,
+        hadoopConf,
+        0.1,
+        true);
+
+    try (HoodieAvroParquetWriter writer = new HoodieAvroParquetWriter(
+        new Path(baseFilePath),
+        parquetConfig,
+        baseInstantTime,
+        new LocalTaskContextSupplier(),
+        true)) {
+      for (IndexedRecord record : records) {
+        writer.writeAvro(
+            (String) record.get(schema.getField(ROW_KEY).pos()), record);
+      }
+    }
+    return new HoodieBaseFile(baseFilePath);
+  }
+
+  public static HoodieLogFile createLogFile(
+      FileSystem fileSystem,
+      String logFilePath,
+      List<IndexedRecord> records,
+      Schema schema,
+      String fileId,
+      String logInstantTime,
+      int version,
+      HoodieLogBlock.HoodieLogBlockType blockType
+  ) throws InterruptedException, IOException {
+    try (HoodieLogFormat.Writer writer =
+             HoodieLogFormat.newWriterBuilder()
+                 .onParentPath(new Path(logFilePath).getParent())
+                 .withFileExtension(HoodieLogFile.DELTA_EXTENSION)
+                 .withFileId(fileId)
+                 .withDeltaCommit(logInstantTime)
+                 .withLogVersion(version)
+                 .withFs(fileSystem).build()) {
+      Map<HoodieLogBlock.HeaderMetadataType, String> header = new HashMap<>();
+      header.put(HoodieLogBlock.HeaderMetadataType.INSTANT_TIME, logInstantTime);
+      header.put(HoodieLogBlock.HeaderMetadataType.SCHEMA, schema.toString());
+      // TODO: use global properties instead.
+      TypedProperties properties = new TypedProperties();
+      properties.setProperty(
+          "hoodie.datasource.write.precombine.field", "timestamp");

Review Comment:
   Yeah, I will add a global var for this. Current the framework focus on one specific table, and schema. We can expand it later if needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1851002116

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413",
       "triggerID" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414",
       "triggerID" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "15f28aa59354f27033840dc140f1e71483257fc4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "15f28aa59354f27033840dc140f1e71483257fc4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9450ab6f8a73add7f5b25b92c73429ce5f3f25ec UNKNOWN
   * d67c76a2f7fa1b448921ad1deb3cf9b36595650a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414) 
   * 15f28aa59354f27033840dc140f1e71483257fc4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1423390827


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/DataGenerationPlan.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import java.util.List;
+
+/**
+ * The blueprint of records that will be generated
+ * by the data generator.
+ *
+ * Current limitations:
+ * 1. One plan generates one file, either a base file, or a log file.
+ * 2. One file contains one operation, e.g., insert, delete, or update.
+ */
+public class DataGenerationPlan {

Review Comment:
   Is this going to be integrated with `HoodieTestTable` or similar to generate test tables without using transactional writes?
   



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieTestReaderContext.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecordMerger;
+import org.apache.hudi.common.model.HoodieEmptyRecord;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieOperation;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.ClosableIterator;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.io.storage.HoodieAvroParquetReader;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.function.UnaryOperator;
+
+import static org.apache.hudi.common.model.HoodieRecordMerger.DEFAULT_MERGER_STRATEGY_UUID;
+import static org.apache.hudi.common.testutils.reader.HoodieFileSliceTestUtils.ROW_KEY;
+
+public class HoodieTestReaderContext extends HoodieReaderContext<IndexedRecord> {
+  @Override
+  public FileSystem getFs(String path, Configuration conf) {
+    return FSUtils.getFs(path, conf);
+  }
+
+  @Override
+  public ClosableIterator<IndexedRecord> getFileRecordIterator(
+      Path filePath,
+      long start,
+      long length,
+      Schema dataSchema,
+      Schema requiredSchema,
+      Configuration conf
+  ) throws IOException {
+    HoodieAvroParquetReader reader = new HoodieAvroParquetReader(conf, filePath);
+    return reader.getIndexedRecordIterator(dataSchema, requiredSchema);
+  }
+
+  @Override
+  public IndexedRecord convertAvroRecord(IndexedRecord record) {
+    return record;
+  }
+
+  @Override
+  public HoodieRecordMerger getRecordMerger(String mergerStrategy) {
+    switch (mergerStrategy) {
+      case DEFAULT_MERGER_STRATEGY_UUID:
+        return new HoodieAvroRecordMerger();
+      default:
+        throw new HoodieException(
+            "The merger strategy UUID is not supported: " + mergerStrategy);
+    }
+  }
+
+  @Override
+  public Object getValue(IndexedRecord record, Schema schema, String fieldName) {
+    return getFieldValueFromIndexedRecord(record, schema, fieldName);
+  }
+
+  @Override
+  public String getRecordKey(IndexedRecord record, Schema schema) {
+    return getFieldValueFromIndexedRecord(record, schema, ROW_KEY).toString();
+  }
+
+  @Override
+  public Comparable getOrderingValue(
+      Option<IndexedRecord> recordOpt,
+      Map<String, Object> metadataMap,
+      Schema schema,
+      TypedProperties props
+  ) {
+    if (metadataMap.containsKey(INTERNAL_META_ORDERING_FIELD)) {
+      return (Comparable) metadataMap.get(INTERNAL_META_ORDERING_FIELD);
+    }
+
+    if (!recordOpt.isPresent()) {
+      return 0;
+    }
+
+    String orderingFieldName = ConfigUtils.getOrderingField(props);
+    Object value = getFieldValueFromIndexedRecord(recordOpt.get(), schema, orderingFieldName);
+    return value != null ? (Comparable) value : 0;
+  }
+
+  @Override
+  public HoodieRecord<IndexedRecord> constructHoodieRecord(
+      Option<IndexedRecord> recordOpt,
+      Map<String, Object> metadataMap
+  ) {
+    if (!recordOpt.isPresent()) {
+      HoodieKey key = new HoodieKey((String) metadataMap.get(INTERNAL_META_RECORD_KEY),
+          (String) metadataMap.get(INTERNAL_META_PARTITION_PATH));
+      return new HoodieEmptyRecord<>(
+          key,
+          HoodieOperation.DELETE,
+          (Comparable<?>) metadataMap.get(INTERNAL_META_ORDERING_FIELD),
+          HoodieRecord.HoodieRecordType.AVRO);
+    }
+    return new HoodieAvroIndexedRecord(recordOpt.get());
+  }
+
+  @Override
+  public IndexedRecord seal(IndexedRecord record) {
+    Schema schema = record.getSchema();
+    GenericRecordBuilder builder = new GenericRecordBuilder(schema);
+    for (Schema.Field field : schema.getFields()) {
+      builder.set(field, record.get(field.pos()));
+    }
+    return builder.build();
+  }
+
+  @Override
+  public ClosableIterator<IndexedRecord> mergeBootstrapReaders(ClosableIterator<IndexedRecord> skeletonFileIterator, ClosableIterator<IndexedRecord> dataFileIterator) {
+    return null;
+  }
+
+  @Override
+  public UnaryOperator<IndexedRecord> projectRecord(Schema from, Schema to) {
+    return null;
+  }

Review Comment:
   Is this going to be implemented in a separate PR?



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {

Review Comment:
   Should this be part of `HoodieTestTable` instead of utils class/methods so one can easily write a Hudi table on storage for testing?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1423158914


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";
+  public static final String PARQUET = ".parquet";
+  public static final String ROW_KEY = "_row_key";
+  public static final String PARTITION_PATH = "partition_path";
+  public static final String RIDER = "rider";
+  public static final String DRIVER = "driver";
+  public static final String TIMESTAMP = "timestamp";
+  public static final HoodieTestDataGenerator DATA_GEN =
+      new HoodieTestDataGenerator(0xDEED);
+
+  // We use a number to represent a record key, and a (start, end) range
+  // to represent a set of record keys between start <= k <= end.
+  public static class KeyRange {
+    public int start;
+    public int end;
+
+    public KeyRange(int start, int end) {
+      this.start = start;
+      this.end = end;
+    }
+  }
+
+  private static Path generateBaseFilePath(
+      String basePath,
+      String fileId,
+      String instantTime
+  ) {
+    return new Path(
+        basePath + FORWARD_SLASH
+        + baseFileName(instantTime, fileId, PARQUET));
+  }
+
+  private static Path generateLogFilePath(
+      String basePath,
+      String fileId,
+      String instantTime,
+      int version) {
+    return new Path(
+        basePath + FORWARD_SLASH + logFileName(
+        instantTime, fileId, version));
+  }
+
+  // Note:
+  // "start < end" means start <= k <= end.
+  // "start == end" means k = start.
+  // "start > end" means no keys.
+  private static List<String> generateKeys(KeyRange range) {
+    List<String> keys = new ArrayList<>();
+    if (range.start == range.end) {
+      keys.add(String.valueOf(range.start));
+    } else {
+      keys = IntStream
+          .rangeClosed(range.start, range.end)
+          .boxed()
+          .map(String::valueOf).collect(Collectors.toList());
+    }
+    return keys;
+  }
+
+  private static List<IndexedRecord> generateRecords(DataGenerationPlan plan) {
+    List<IndexedRecord> records = new ArrayList<>();
+    List<String> keys = plan.getRecordKeys();
+    for (String key : keys) {
+      records.add(DATA_GEN.generateGenericRecord(
+          key,
+          plan.getPartitionPath(),
+          RIDER + PERIOD + UUID.randomUUID().toString(),
+          DRIVER + PERIOD + UUID.randomUUID().toString(),
+          plan.getTimestamp(),
+          plan.getOperationType() == DELETE,
+          false
+      ));
+    }
+    return records;
+  }
+
+  private static HoodieDataBlock getDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path logFilePath
+  ) {
+    return createDataBlock(
+        dataBlockType,
+        records.stream().map(HoodieAvroIndexedRecord::new)
+            .collect(Collectors.toList()),
+        header,
+        logFilePath);
+  }
+
+  private static HoodieDataBlock createDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<HoodieRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path pathForReader
+  ) {
+    switch (dataBlockType) {
+      case CDC_DATA_BLOCK:
+        return new HoodieCDCDataBlock(
+            records,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case AVRO_DATA_BLOCK:
+        return new HoodieAvroDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case HFILE_DATA_BLOCK:
+        return new HoodieHFileDataBlock(
+            records,
+            header,
+            Compression.Algorithm.GZ,
+            pathForReader);
+      case PARQUET_DATA_BLOCK:
+        return new HoodieParquetDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD,
+            CompressionCodecName.GZIP,
+            0.1,
+            true);
+      default:
+        throw new RuntimeException(
+            "Unknown data block type " + dataBlockType);
+    }
+  }
+
+  public static HoodieDeleteBlock getDeleteBlock(
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Schema schema,
+      Properties props
+  ) {
+    List<HoodieRecord> hoodieRecords = records.stream()
+        .map(r -> {
+          String rowKey = (String) r.get(r.getSchema().getField(ROW_KEY).pos());
+          String partitionPath = (String) r.get(r.getSchema().getField(PARTITION_PATH).pos());
+          return new HoodieAvroIndexedRecord(new HoodieKey(rowKey, partitionPath), r);
+        })
+        .collect(Collectors.toList());
+    return new HoodieDeleteBlock(
+        hoodieRecords.stream().map(
+            r -> Pair.of(DeleteRecord.create(
+                r.getKey(), r.getOrderingValue(schema, props)), -1L))
+            .collect(Collectors.toList()),
+        false,
+        header
+    );
+  }
+
+  public static HoodieBaseFile createBaseFile(
+      String baseFilePath,
+      List<IndexedRecord> records,
+      Schema schema,
+      String baseInstantTime
+  ) throws IOException {
+    Configuration hadoopConf = new Configuration();
+
+    BloomFilter filter = BloomFilterFactory.createBloomFilter(
+        1000,
+        0.0001,
+        10000,
+        BloomFilterTypeCode.DYNAMIC_V0.name());
+    HoodieAvroWriteSupport<IndexedRecord> writeSupport = new HoodieAvroWriteSupport<>(
+        new AvroSchemaConverter().convert(schema),
+        schema,
+        Option.of(filter),
+        new Properties());
+    HoodieParquetConfig<HoodieAvroWriteSupport> parquetConfig = new HoodieParquetConfig(
+        writeSupport,
+        CompressionCodecName.GZIP,
+        ParquetWriter.DEFAULT_BLOCK_SIZE,
+        ParquetWriter.DEFAULT_PAGE_SIZE,
+        1024 * 1024 * 1024,
+        hadoopConf,
+        0.1,
+        true);
+

Review Comment:
   Added a task to track this problem: [HUDI-7214](https://issues.apache.org/jira/browse/HUDI-7214)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1420923427


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/DataGenerationPlan.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import java.util.List;
+
+/**
+ * The blueprint of records that will be generated
+ * by the data generator.
+ */
+public class DataGenerationPlan {
+  // The values for "_row_key" field.
+  private final List<String> recordKeys;

Review Comment:
   If needed in the future, we can add that functionality.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176][WIP] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1843609890

   @yihua , @codope , @danny0405


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1420927005


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";
+  public static final String PARQUET = ".parquet";
+  public static final String ROW_KEY = "_row_key";
+  public static final String PARTITION_PATH = "partition_path";
+  public static final String RIDER = "rider";
+  public static final String DRIVER = "driver";
+  public static final String TIMESTAMP = "timestamp";
+  public static final HoodieTestDataGenerator DATA_GEN =
+      new HoodieTestDataGenerator(0xDEED);
+
+  // We use a number to represent a record key, and a (start, end) range
+  // to represent a set of record keys between start <= k <= end.
+  public static class KeyRange {
+    public int start;
+    public int end;
+
+    public KeyRange(int start, int end) {
+      this.start = start;
+      this.end = end;
+    }
+  }
+
+  private static Path generateBaseFilePath(
+      String basePath,
+      String fileId,
+      String instantTime
+  ) {
+    return new Path(
+        basePath + FORWARD_SLASH
+        + baseFileName(instantTime, fileId, PARQUET));
+  }
+
+  private static Path generateLogFilePath(
+      String basePath,
+      String fileId,
+      String instantTime,
+      int version) {
+    return new Path(
+        basePath + FORWARD_SLASH + logFileName(
+        instantTime, fileId, version));
+  }
+
+  // Note:
+  // "start < end" means start <= k <= end.
+  // "start == end" means k = start.
+  // "start > end" means no keys.
+  private static List<String> generateKeys(KeyRange range) {
+    List<String> keys = new ArrayList<>();
+    if (range.start == range.end) {
+      keys.add(String.valueOf(range.start));
+    } else {
+      keys = IntStream
+          .rangeClosed(range.start, range.end)
+          .boxed()
+          .map(String::valueOf).collect(Collectors.toList());
+    }
+    return keys;
+  }
+
+  private static List<IndexedRecord> generateRecords(DataGenerationPlan plan) {
+    List<IndexedRecord> records = new ArrayList<>();
+    List<String> keys = plan.getRecordKeys();
+    for (String key : keys) {
+      records.add(DATA_GEN.generateGenericRecord(
+          key,
+          plan.getPartitionPath(),
+          RIDER + PERIOD + UUID.randomUUID().toString(),
+          DRIVER + PERIOD + UUID.randomUUID().toString(),
+          plan.getTimestamp(),
+          plan.getOperationType() == DELETE,
+          false
+      ));
+    }
+    return records;
+  }
+
+  private static HoodieDataBlock getDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path logFilePath
+  ) {
+    return createDataBlock(
+        dataBlockType,
+        records.stream().map(HoodieAvroIndexedRecord::new)
+            .collect(Collectors.toList()),
+        header,
+        logFilePath);
+  }
+
+  private static HoodieDataBlock createDataBlock(
+      HoodieLogBlock.HoodieLogBlockType dataBlockType,
+      List<HoodieRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Path pathForReader
+  ) {
+    switch (dataBlockType) {
+      case CDC_DATA_BLOCK:
+        return new HoodieCDCDataBlock(
+            records,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case AVRO_DATA_BLOCK:
+        return new HoodieAvroDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD);
+      case HFILE_DATA_BLOCK:
+        return new HoodieHFileDataBlock(
+            records,
+            header,
+            Compression.Algorithm.GZ,
+            pathForReader);
+      case PARQUET_DATA_BLOCK:
+        return new HoodieParquetDataBlock(
+            records,
+            false,
+            header,
+            HoodieRecord.RECORD_KEY_METADATA_FIELD,
+            CompressionCodecName.GZIP,
+            0.1,
+            true);
+      default:
+        throw new RuntimeException(
+            "Unknown data block type " + dataBlockType);
+    }
+  }
+
+  public static HoodieDeleteBlock getDeleteBlock(
+      List<IndexedRecord> records,
+      Map<HoodieLogBlock.HeaderMetadataType, String> header,
+      Schema schema,
+      Properties props
+  ) {
+    List<HoodieRecord> hoodieRecords = records.stream()
+        .map(r -> {
+          String rowKey = (String) r.get(r.getSchema().getField(ROW_KEY).pos());
+          String partitionPath = (String) r.get(r.getSchema().getField(PARTITION_PATH).pos());
+          return new HoodieAvroIndexedRecord(new HoodieKey(rowKey, partitionPath), r);
+        })
+        .collect(Collectors.toList());
+    return new HoodieDeleteBlock(
+        hoodieRecords.stream().map(
+            r -> Pair.of(DeleteRecord.create(
+                r.getKey(), r.getOrderingValue(schema, props)), -1L))
+            .collect(Collectors.toList()),
+        false,
+        header
+    );
+  }
+
+  public static HoodieBaseFile createBaseFile(
+      String baseFilePath,
+      List<IndexedRecord> records,
+      Schema schema,
+      String baseInstantTime
+  ) throws IOException {
+    Configuration hadoopConf = new Configuration();
+
+    BloomFilter filter = BloomFilterFactory.createBloomFilter(
+        1000,
+        0.0001,
+        10000,
+        BloomFilterTypeCode.DYNAMIC_V0.name());
+    HoodieAvroWriteSupport<IndexedRecord> writeSupport = new HoodieAvroWriteSupport<>(
+        new AvroSchemaConverter().convert(schema),
+        schema,
+        Option.of(filter),
+        new Properties());
+    HoodieParquetConfig<HoodieAvroWriteSupport> parquetConfig = new HoodieParquetConfig(
+        writeSupport,
+        CompressionCodecName.GZIP,
+        ParquetWriter.DEFAULT_BLOCK_SIZE,
+        ParquetWriter.DEFAULT_PAGE_SIZE,
+        1024 * 1024 * 1024,
+        hadoopConf,
+        0.1,
+        true);
+

Review Comment:
   I copied from a test case. I will figure it out later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1847697329

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413",
       "triggerID" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1847706543

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413",
       "triggerID" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413) 
   * 9450ab6f8a73add7f5b25b92c73429ce5f3f25ec UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1420926421


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";
+  public static final String PARQUET = ".parquet";
+  public static final String ROW_KEY = "_row_key";
+  public static final String PARTITION_PATH = "partition_path";
+  public static final String RIDER = "rider";
+  public static final String DRIVER = "driver";
+  public static final String TIMESTAMP = "timestamp";
+  public static final HoodieTestDataGenerator DATA_GEN =
+      new HoodieTestDataGenerator(0xDEED);
+
+  // We use a number to represent a record key, and a (start, end) range
+  // to represent a set of record keys between start <= k <= end.
+  public static class KeyRange {
+    public int start;
+    public int end;
+
+    public KeyRange(int start, int end) {
+      this.start = start;
+      this.end = end;
+    }
+  }
+
+  private static Path generateBaseFilePath(
+      String basePath,
+      String fileId,
+      String instantTime
+  ) {
+    return new Path(
+        basePath + FORWARD_SLASH
+        + baseFileName(instantTime, fileId, PARQUET));
+  }
+
+  private static Path generateLogFilePath(
+      String basePath,
+      String fileId,
+      String instantTime,
+      int version) {
+    return new Path(
+        basePath + FORWARD_SLASH + logFileName(
+        instantTime, fileId, version));
+  }
+
+  // Note:
+  // "start < end" means start <= k <= end.
+  // "start == end" means k = start.
+  // "start > end" means no keys.
+  private static List<String> generateKeys(KeyRange range) {
+    List<String> keys = new ArrayList<>();
+    if (range.start == range.end) {
+      keys.add(String.valueOf(range.start));
+    } else {
+      keys = IntStream
+          .rangeClosed(range.start, range.end)
+          .boxed()
+          .map(String::valueOf).collect(Collectors.toList());
+    }
+    return keys;
+  }
+
+  private static List<IndexedRecord> generateRecords(DataGenerationPlan plan) {
+    List<IndexedRecord> records = new ArrayList<>();
+    List<String> keys = plan.getRecordKeys();
+    for (String key : keys) {
+      records.add(DATA_GEN.generateGenericRecord(

Review Comment:
   Honestly datagenerator itself is already too hard to understand. I want to isolate the logic for the framework right now. Later when we try to import more test cases, and we see more general requests, and then I will add them to the data gen. Even we should refactor it a bit more.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1420928351


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";

Review Comment:
   Do you mean why var for these literals?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1847926504

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413",
       "triggerID" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414",
       "triggerID" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413) 
   * 9450ab6f8a73add7f5b25b92c73429ce5f3f25ec UNKNOWN
   * d67c76a2f7fa1b448921ad1deb3cf9b36595650a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1424578340


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/DataGenerationPlan.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import java.util.List;
+
+/**
+ * The blueprint of records that will be generated
+ * by the data generator.
+ *
+ * Current limitations:
+ * 1. One plan generates one file, either a base file, or a log file.
+ * 2. One file contains one operation, e.g., insert, delete, or update.
+ */
+public class DataGenerationPlan {

Review Comment:
   Let's have only one test framework to generate test tables.  Let's do that in the follow-up PR.  Also add a guideline on how to use this new test framework to construct tests that needs preparation of a Hudi table with data files.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1847792258

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413",
       "triggerID" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414",
       "triggerID" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413) 
   * 9450ab6f8a73add7f5b25b92c73429ce5f3f25ec UNKNOWN
   * d67c76a2f7fa1b448921ad1deb3cf9b36595650a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1851206311

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21413",
       "triggerID" : "d8ab3f6229d436bcfa21c48e59c68e6c3a8f7c92",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9450ab6f8a73add7f5b25b92c73429ce5f3f25ec",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21414",
       "triggerID" : "d67c76a2f7fa1b448921ad1deb3cf9b36595650a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "15f28aa59354f27033840dc140f1e71483257fc4",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21455",
       "triggerID" : "15f28aa59354f27033840dc140f1e71483257fc4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9450ab6f8a73add7f5b25b92c73429ce5f3f25ec UNKNOWN
   * 15f28aa59354f27033840dc140f1e71483257fc4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21455) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1423170204


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieFileSliceTestUtils.java:
##########
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import org.apache.hudi.avro.HoodieAvroWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.bloom.BloomFilterTypeCode;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.LocalTaskContextSupplier;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieAvroIndexedRecord;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieLogFile;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.log.HoodieLogFormat;
+import org.apache.hudi.common.table.log.block.HoodieAvroDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieCDCDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.table.log.block.HoodieHFileDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieLogBlock;
+import org.apache.hudi.common.table.log.block.HoodieParquetDataBlock;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.io.storage.HoodieAvroParquetWriter;
+import org.apache.hudi.io.storage.HoodieParquetConfig;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.io.compress.Compression;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.DELETE_BLOCK;
+import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HoodieLogBlockType.PARQUET_DATA_BLOCK;
+import static org.apache.hudi.common.testutils.FileCreateUtils.baseFileName;
+import static org.apache.hudi.common.testutils.FileCreateUtils.logFileName;
+import static org.apache.hudi.common.testutils.HoodieTestDataGenerator.AVRO_SCHEMA;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.DELETE;
+import static org.apache.hudi.common.testutils.reader.DataGenerationPlan.OperationType.INSERT;
+
+public class HoodieFileSliceTestUtils {
+  public static final String FORWARD_SLASH = "/";
+  public static final String PERIOD = ".";

Review Comment:
   Just a style. Here we can either use the literals directly or use variables. :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10263:
URL: https://github.com/apache/hudi/pull/10263#discussion_r1424490970


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/DataGenerationPlan.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.testutils.reader;
+
+import java.util.List;
+
+/**
+ * The blueprint of records that will be generated
+ * by the data generator.
+ *
+ * Current limitations:
+ * 1. One plan generates one file, either a base file, or a log file.
+ * 2. One file contains one operation, e.g., insert, delete, or update.
+ */
+public class DataGenerationPlan {

Review Comment:
   So far I haven't seen any necessity for it. But in the future if we need, we can extend it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7176] Add file group reader test framework [hudi]

Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #10263:
URL: https://github.com/apache/hudi/pull/10263#issuecomment-1852689024

   @yihua, I have addressed your comments. My overall plan is to land this PR, and use another PR to further simplify this framework and integrate more with other existing modules. How does it sound?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org