You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by vh...@apache.org on 2025/09/23 16:20:51 UTC
(hudi) branch minor-add-configurable-hfile-config updated (634d72e29d66 -> 9b3a60b5b74f)
This is an automated email from the ASF dual-hosted git repository.
vhs pushed a change to branch minor-add-configurable-hfile-config
in repository https://gitbox.apache.org/repos/asf/hudi.git
discard 634d72e29d66 Ensure proper wiring in write and read paths
discard cb7123487dff Fix scalastyle issue
omit 1efd26a14938 Fix config usage typo
omit ddf7f9a9c6a6 Minor add configurable hfile config
add 4cefaa226d0f chore: Update PR template and compliance check after deprecating JIRA (#13883)
add 2efb90c0a7e9 feat(merger): [RFC-101] Remove shouldFlush and outer merge from record merger interface (#13904)
add 888263fad601 chore: update issue templates post JIRA migration (#13907)
add be2814511c1f chore: Change issue template to use `value` instead of `placeholder` (#13913)
add 9031935b3bf8 chore: change label type for community support issue (#13920)
add 042a0c3b8ec0 [MINOR] Reducing few noisy logs (#13895)
add b4cf65e20c67 [HUDI-9731] Add index definition for Record level Index (#13741)
add a9c101125948 fix: adding back PRECOMBINE field Config property in DataSourceReadOptions (#13900)
add ff00d212c404 feat(merger): [RFC-101] Update HoodieRecordMerger interface and implementations (#13742)
add 6916c309cef9 [HUDI-9666] Fix the record key encoding with a single record key field and add a guard for complex key generator (#13650)
add 36619c8a2ae2 [HUDI-9788] Fix the incremental query when records from multi commits are from the same file (#13840)
add e3e8b7347940 [HUDI-4985] Update ARM64 Docker demo images to use latest Hudi version (#13902)
add 7a7ad4b35333 fix: [HUDI-9611] Metadata writer tries to initialize invalid partitions (#13583)
add f64cd6655f73 perf(spark): Write handle perf fix for prepending meta fields (#13860)
add c50f5604e8b8 fix(storage-format)!: Fix HFile writer to properly encode checksum for compatibility (#13897)
add 1e0513d1585a [MINOR] Incremental and Time Travel Query Testing (#13934)
add c2b564dbbe42 [HUDI-9778] Remove HoodieRecord wrapper from MDT reads (#13796)
add 50a75f27ba69 fix(ingest): HudiIncrSource - Handle nullable partition filter (#13945)
add 99e04da76c7a perf: Update the hfile prefetch size threshold to 50 (#13946)
add 76bb580ae149 [HUDI-7915] Spark 4 support (#12772)
add 0e30bd69d82b feat(merger): Payload deprecation bug fixes (#13947)
add 2817ccf8a31f test: fix flink flaky test TestStreamReadMonitoringFunction#testCheckpointRestoreWithLimit (#13936)
add 66d2780d8983 [HUDI-9777] Avoid redundant file length fetch in HoodieNativeHFileReader (#13797)
add b3b4658709d2 fix: issue when reading a Hudi table with nested partition fields (#13943)
add 24f1be9a7070 [HUDI-9776] Reduce redundant config lookups and minor FSView optimizations (#13803)
add 19961e77d539 feat(merger): Make HoodieRecord delete handling consistent with merger code path (#13951)
add bf50dd438fe3 [HUDI-9158] Rename heartbeat poll secs to lock renewal secs (#13968)
add 9a5467afe9d8 perf: Avoid repeated work in HFile reads and writes (#13962)
add fb4dc84faa47 refactor: Move FlinkRecordContext to hudi-flink-client module (#13958)
add d10310f4d9f3 fix: fixing hivesync tool in docker (#13957)
add be6f08d56643 feat(metadata): Improve Logical Type Handling on Col Stats (#13711)
add 0685e8bbebee [HUDI-9757] Support TLS authentication for Datahub sync (#13861)
add 6377edb97a6e fix: Fixing schema for Merge Into writes for COW (#13967)
add ff452bb1dd23 Minor add configurable hfile config
add 9244d706b5a6 Fix config usage typo
add ae8c9b304f2b Fix scalastyle issue
add 9b3a60b5b74f Ensure proper wiring in write and read paths
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (634d72e29d66)
\
N -- N -- N refs/heads/minor-add-configurable-hfile-config (9b3a60b5b74f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/ISSUE_TEMPLATE/SUPPORT_REQUEST.md | 56 --
.github/ISSUE_TEMPLATE/config.yml | 14 +-
.github/ISSUE_TEMPLATE/hudi_bug.yml | 58 ++
.github/ISSUE_TEMPLATE/hudi_community_support.yml | 88 +++
.github/ISSUE_TEMPLATE/hudi_feature.yml | 61 ++
.github/ISSUE_TEMPLATE/hudi_improvement.yml | 59 ++
.github/PULL_REQUEST_TEMPLATE.md | 31 +-
.github/workflows/bot.yml | 95 ++-
.github/workflows/pr_compliance.yml | 9 +-
.github/workflows/pr_title_validation.yml | 68 +++
.github/workflows/update_pr_compliance.yml | 9 +-
azure-pipelines-20230430.yml | 2 +
...r-compose_hadoop334_hive313_spark353_arm64.yml} | 194 +++---
docker/hoodie/hadoop/base/Dockerfile | 4 +-
docker/hoodie/hadoop/datanode/Dockerfile | 2 +-
docker/hoodie/hadoop/historyserver/Dockerfile | 17 +-
docker/hoodie/hadoop/hive_base/Dockerfile | 5 +-
docker/hoodie/hadoop/hive_base/conf/hive-env.sh | 2 +-
docker/hoodie/hadoop/hive_base/conf/hive-site.xml | 46 +-
docker/hoodie/hadoop/namenode/Dockerfile | 2 +-
docker/hoodie/hadoop/spark_base/Dockerfile | 29 +-
docker/hoodie/hadoop/sparkadhoc/Dockerfile | 10 +-
docker/hoodie/hadoop/sparkmaster/Dockerfile | 4 +-
docker/hoodie/hadoop/sparkworker/Dockerfile | 4 +-
docker/setup_demo.sh | 4 +-
docker/stop_demo.sh | 4 +-
hudi-aws/pom.xml | 4 +
.../org/apache/hudi/cli/commands/SparkMain.java | 6 +-
.../hudi/cli/utils/SparkTempViewProvider.java | 2 +-
.../cli/functional/CLIFunctionalTestHarness.java | 3 +-
.../apache/hudi/client/BaseHoodieWriteClient.java | 9 +-
.../transaction/lock/StorageBasedLockProvider.java | 2 +-
.../org/apache/hudi/config/HoodieWriteConfig.java | 40 ++
.../apache/hudi/config/StorageBasedLockConfig.java | 17 +-
.../org/apache/hudi/index/HoodieIndexUtils.java | 37 +-
.../apache/hudi/index/bloom/HoodieBloomIndex.java | 7 +-
.../java/org/apache/hudi/io/BaseCreateHandle.java | 2 +-
.../hudi/io/FileGroupReaderBasedMergeHandle.java | 10 -
.../org/apache/hudi/io/HoodieAppendHandle.java | 21 +-
.../org/apache/hudi/io/HoodieBinaryCopyHandle.java | 7 +-
.../hudi/io/HoodieMergeHandleWithChangeLog.java | 11 +-
.../apache/hudi/io/HoodieSortedMergeHandle.java | 6 +-
.../io/HoodieSortedMergeHandleWithChangeLog.java | 6 +-
.../java/org/apache/hudi/io/HoodieWriteHandle.java | 9 +-
.../org/apache/hudi/io/HoodieWriteMergeHandle.java | 84 +--
.../hudi/io/SecondaryIndexStreamingTracker.java | 11 +-
.../hudi/keygen/ComplexAvroKeyGenerator.java | 18 +-
.../java/org/apache/hudi/keygen/KeyGenUtils.java | 35 ++
.../metadata/HoodieBackedTableMetadataWriter.java | 19 +-
.../hudi/table/action/commit/BaseWriteHelper.java | 14 +-
.../table/action/commit/HoodieWriteHelper.java | 4 +-
.../table/upgrade/EightToNineUpgradeHandler.java | 39 +-
.../upgrade/EightToSevenDowngradeHandler.java | 6 +
.../table/upgrade/NineToEightDowngradeHandler.java | 23 +-
.../table/upgrade/SevenToEightUpgradeHandler.java | 7 +
.../hudi/table/upgrade/UpgradeDowngradeUtils.java | 19 +-
.../hudi/client/TestBaseHoodieWriteClient.java | 151 +++++
.../lock/StorageBasedLockProviderTestBase.java | 2 +-
.../lock/TestStorageBasedLockProvider.java | 12 +-
.../hudi/config/TestStorageBasedLockConfig.java | 14 +-
.../org/apache/hudi/io/TestHoodieWriteHandle.java | 2 +-
.../io/TestSecondaryIndexStreamingTracker.java | 25 +-
.../hudi/keygen/TestComplexAvroKeyGenerator.java | 62 +-
.../org/apache/hudi/keygen/TestKeyGenUtils.java | 115 ++++
.../upgrade/TestEightToNineUpgradeHandler.java | 110 ++--
.../upgrade/TestNineToEightDowngradeHandler.java | 27 +-
.../java/org/apache/hudi/testutils/Assertions.java | 9 +
.../hudi/testutils/HoodieMergeOnReadTestUtils.java | 4 +-
.../hudi/utils/HoodieWriterClientTestHarness.java | 3 +-
hudi-client/hudi-flink-client/pom.xml | 4 +
.../client/model/CommitTimeFlinkRecordMerger.java | 18 +-
.../client/model/EventTimeFlinkRecordMerger.java | 36 +-
.../hudi/client/model/HoodieFlinkRecord.java | 18 +-
.../model/PartialUpdateFlinkRecordMerger.java | 61 +-
.../FlinkIncrementalMergeHandleWithChangeLog.java | 9 +-
.../hudi/io/FlinkMergeHandleWithChangeLog.java | 8 +-
.../io/log/block/HoodieFlinkParquetDataBlock.java | 7 +-
.../apache/hudi/io/v2/RowDataLogWriteHandle.java | 14 +-
.../hudi/table/action/commit/FlinkWriteHelper.java | 4 +-
.../hudi/table/format/FlinkRecordContext.java | 4 +-
.../hudi/util/RecordKeyToRowDataConverter.java | 0
.../hudi/merge/TestHoodieFlinkRecordMerger.java | 78 +--
.../merge/TestPartialUpdateFlinkRecordMerger.java | 107 ++--
hudi-client/hudi-java-client/pom.xml | 4 +
.../hudi/table/action/commit/JavaWriteHelper.java | 4 +-
hudi-client/hudi-spark-client/pom.xml | 6 +
.../org/apache/hudi/DefaultSparkRecordMerger.java | 58 +-
.../org/apache/hudi/HoodieSparkRecordMerger.java | 44 --
.../hudi/OverwriteWithLatestSparkRecordMerger.java | 24 +-
.../HoodieSparkBootstrapSchemaProvider.java | 2 +-
.../MultipleSparkJobExecutionStrategy.java | 7 +-
.../client/common/SparkReaderContextFactory.java | 2 +-
.../hudi/client/model/HoodieInternalRow.java | 40 +-
.../client/utils/SparkInternalSchemaConverter.java | 11 +-
.../client/utils/SparkMetadataWriterUtils.java | 38 +-
.../hudi/client/utils/SparkPartitionUtils.java | 4 -
.../hudi/client/utils/SparkValidatorUtils.java | 2 +-
.../validator/SqlQueryPreCommitValidator.java | 2 +-
.../hudi/common/model/HoodieSparkRecord.java | 36 +-
.../bulkinsert/RDDBucketIndexPartitioner.java | 11 +-
.../RDDCustomColumnsSortPartitioner.java | 8 +-
.../apache/hudi/index/HoodieSparkIndexClient.java | 28 +-
.../hudi/index/SparkMetadataTableRecordIndex.java | 5 +-
.../hudi/io/storage/row/HoodieRowCreateHandle.java | 14 +-
.../storage/row/HoodieRowParquetWriteSupport.java | 14 +-
.../apache/hudi/keygen/BuiltinKeyGenerator.java | 40 +-
.../apache/hudi/keygen/ComplexKeyGenerator.java | 8 +-
.../hudi/keygen/GlobalDeleteKeyGenerator.java | 4 +-
.../apache/hudi/merge/SparkRecordMergingUtils.java | 42 +-
.../SparkHoodieBackedTableMetadataWriter.java | 2 +-
.../apache/hudi/stats/SparkValueMetadataUtils.java | 157 +++++
.../bootstrap/ParquetBootstrapMetadataHandler.java | 4 +-
.../commit/BaseSparkCommitActionExecutor.java | 6 +-
.../org/apache/hudi/AvroConversionUtils.scala | 15 +-
.../hudi/BaseSparkInternalRecordContext.java | 68 ++-
.../hudi/HoodieDatasetBulkInsertHelper.scala | 58 +-
.../hudi/HoodiePartitionCDCFileGroupMapping.scala | 9 +-
.../hudi/HoodiePartitionFileSliceMapping.scala | 10 +-
.../HoodiePartitionValues.scala} | 12 +-
.../scala/org/apache/hudi/HoodieSparkUtils.scala | 110 +++-
.../scala/org/apache/hudi/HoodieUTF8String.scala | 28 +-
.../org/apache/hudi/SparkAdapterSupport.scala | 4 +-
.../org/apache/hudi/SparkConversionUtils.scala | 4 +-
.../SparkFileFormatInternalRecordContext.scala | 13 +-
.../SparkFileFormatInternalRowReaderContext.scala | 2 +-
.../scala/org/apache/hudi/SparkJdbcUtils.scala | 64 --
.../main/scala/org/apache/hudi/SparkRowSerDe.scala | 6 +-
.../apache/hudi/spark/internal/ReflectUtil.java | 0
.../apache/spark/HoodieSparkKryoRegistrar.scala | 2 +-
.../apache/spark/sql/BucketPartitionUtils.scala | 5 +-
.../scala/org/apache/spark/sql/DataFrameUtil.scala | 13 +-
.../org/apache/spark/sql/HoodieSchemaUtils.scala | 18 +
.../apache/spark/sql/HoodieUTF8StringFactory.scala | 20 +-
.../org/apache/spark/sql/HoodieUnsafeUtils.scala | 51 +-
.../datasources/SparkParsePartitionUtil.scala | 265 +++++++-
.../org/apache/spark/sql/hudi/SparkAdapter.scala | 167 ++++--
.../spark/sql/hudi/execution/RangeSample.scala | 6 +-
.../org/apache/hudi/client/TestWriteStatus.java | 5 +-
.../common/TestSparkReaderContextFactory.java | 4 +-
.../functional/TestExternalPathHandling.java | 7 +-
.../functional/TestHoodieBackedTableMetadata.java | 43 +-
.../hudi/keygen/KeyGeneratorTestUtilities.java | 200 +++++++
.../hudi/keygen/TestComplexKeyGenerator.java | 226 +++++++
.../apache/hudi/keygen/TestCustomKeyGenerator.java | 165 +++--
.../keygen/TestGlobalDeleteRecordGenerator.java | 81 +--
.../keygen/TestNonpartitionedKeyGenerator.java | 96 +--
.../apache/hudi/keygen/TestSimpleKeyGenerator.java | 56 +-
.../keygen/TestTimestampBasedKeyGenerator.java | 129 ++--
.../TestCreateKeyGeneratorByTypeWithFactory.java | 24 +-
.../TestHoodieSparkKeyGeneratorFactory.java | 17 +-
.../commit/TestCopyOnWriteActionExecutor.java | 22 +-
.../hudi/testutils/DisableOnSpark4Condition.java | 20 +-
.../apache/hudi/testutils/DisabledOnSpark4.java | 18 +-
.../testutils/HoodieSparkClientTestHarness.java | 2 +-
.../testutils/HoodieSparkDeleteRecordMerger.java | 13 +-
.../SparkClientFunctionalTestHarness.java | 2 +-
hudi-common/src/main/avro/HoodieMetadata.avsc | 32 +-
.../main/java/org/apache/hudi/ParquetAdapter.java | 31 +-
.../org/apache/hudi/avro/AvroRecordContext.java | 4 +-
.../java/org/apache/hudi/avro/AvroSchemaUtils.java | 55 +-
.../apache/hudi/avro/ConvertingGenericData.java | 24 +-
.../apache/hudi/avro/HoodieAvroReaderContext.java | 63 +-
.../java/org/apache/hudi/avro/HoodieAvroUtils.java | 147 ++++-
.../apache/hudi/avro/HoodieAvroWrapperUtils.java | 154 ++++-
.../hudi/avro/HoodieBloomFilterWriteSupport.java | 8 +-
.../org/apache/hudi/avro/JoinedGenericRecord.java | 90 +++
.../org/apache/hudi/avro/processors/Parser.java | 10 +-
.../apache/hudi/common/config/ConfigProperty.java | 40 +-
.../hudi/common/config/HoodieIndexingConfig.java | 3 +-
.../hudi/common/config/HoodieMetadataConfig.java | 7 +-
.../hudi/common/engine/HoodieReaderContext.java | 2 +-
.../apache/hudi/common/engine/RecordContext.java | 19 +-
.../java/org/apache/hudi/common/fs/FSUtils.java | 27 +-
.../common/model/DefaultHoodieRecordPayload.java | 24 +-
.../hudi/common/model/EventTimeAvroPayload.java | 11 +-
.../hudi/common/model/HoodieAvroIndexedRecord.java | 53 +-
.../apache/hudi/common/model/HoodieAvroRecord.java | 16 +-
.../hudi/common/model/HoodieAvroRecordMerger.java | 82 ++-
.../hudi/common/model/HoodieEmptyRecord.java | 3 +-
.../apache/hudi/common/model/HoodieLogFile.java | 2 +-
.../common/model/HoodieMetadataRecordMerger.java | 51 --
.../model/HoodiePreCombineAvroRecordMerger.java | 44 +-
.../org/apache/hudi/common/model/HoodieRecord.java | 9 +-
.../hudi/common/model/HoodieRecordMerger.java | 67 ++-
.../apache/hudi/common/model/HoodieWriteStat.java | 1 +
.../common/model/OverwriteWithLatestMerger.java | 10 +-
.../hudi/common/table/HoodieTableConfig.java | 23 +-
.../hudi/common/table/HoodieTableMetaClient.java | 80 ++-
.../table/log/HoodieMergedLogRecordScanner.java | 22 +-
.../table/log/HoodieUnMergedLogRecordScanner.java | 2 +-
.../table/read/BufferedRecordMergerFactory.java | 239 ++------
.../hudi/common/table/read/BufferedRecords.java | 13 +-
.../hudi/common/table/read/DeleteContext.java | 21 +-
.../table/read/FileGroupReaderSchemaHandler.java | 3 +-
.../common/table/read/HoodieFileGroupReader.java | 7 +-
.../table/read/buffer/FileGroupRecordBuffer.java | 2 +-
.../read/buffer/KeyBasedFileGroupRecordBuffer.java | 1 -
.../buffer/PositionBasedFileGroupRecordBuffer.java | 3 +-
.../StreamingFileGroupRecordBufferLoader.java | 15 +-
.../table/view/AbstractTableFileSystemView.java | 12 +-
.../org/apache/hudi/common/util/ConfigUtils.java | 7 +-
.../org/apache/hudi/common/util/DateTimeUtils.java | 44 ++
.../apache/hudi/common/util/FileFormatUtils.java | 30 +-
.../org/apache/hudi/common/util/HFileUtils.java | 21 +-
.../apache/hudi/common/util/HoodieDataUtils.java | 27 +
.../apache/hudi/common/util/HoodieRecordUtils.java | 13 +-
.../org/apache/hudi/common/util/SortUtils.java | 36 ++
.../org/apache/hudi/expression/Comparators.java | 3 +
.../record/HoodieRecordIndex.java} | 22 +-
.../java/org/apache/hudi/internal/schema/Type.java | 11 +-
.../org/apache/hudi/internal/schema/Types.java | 211 ++++++-
.../convert/AvroInternalSchemaConverter.java | 69 ++-
.../hudi/internal/schema/utils/Conversions.java | 2 +
.../internal/schema/utils/SchemaChangeUtils.java | 79 ++-
.../hudi/internal/schema/utils/SerDeHelper.java | 25 +
.../io/storage/ColumnRangeMetadataProvider.java | 5 +-
.../apache/hudi/io/storage/HFileReaderFactory.java | 27 +-
.../io/storage/HoodieAvroHFileReaderImplBase.java | 41 +-
.../hudi/io/storage/HoodieFileReaderFactory.java | 20 +
.../io/storage/HoodieNativeAvroHFileReader.java | 43 +-
.../java/org/apache/hudi/keygen/KeyGenerator.java | 7 -
.../hudi/keygen/constant/KeyGeneratorType.java | 16 +
.../apache/hudi/metadata/BaseTableMetadata.java | 57 +-
.../hudi/metadata/HoodieBackedTableMetadata.java | 38 +-
.../apache/hudi/metadata/HoodieIndexVersion.java | 49 +-
.../hudi/metadata/HoodieMetadataPayload.java | 16 +-
.../hudi/metadata/HoodieTableMetadataUtil.java | 350 +++++++----
.../hudi/metadata/MetadataPartitionType.java | 32 +-
.../SecondaryIndexRecordGenerationUtils.java | 4 +-
.../model => stats}/HoodieColumnRangeMetadata.java | 73 ++-
.../java/org/apache/hudi/stats/ValueMetadata.java | 266 +++++++++
.../main/java/org/apache/hudi/stats/ValueType.java | 665 +++++++++++++++++++++
.../parquet/schema/LogicalTypeParquetAdapter.java | 92 +++
.../parquet/schema/OriginalTypeParquetAdapter.java | 83 +++
.../org/apache/hudi/avro/TestHoodieAvroUtils.java | 51 ++
.../hudi/common/config/TestConfigProperty.java | 12 +
.../model/TestDefaultHoodieRecordPayload.java | 18 +-
.../common/model/TestHoodieCommitMetadata.java | 3 +-
.../model/TestHoodieMetadataRecordMerger.java | 65 --
.../hudi/common/model/TestHoodieWriteStat.java | 4 +-
.../common/table/read/SchemaHandlerTestBase.java | 2 +-
.../read/buffer/BaseTestFileGroupRecordBuffer.java | 29 +-
.../versioning/BaseTestCommitMetadataSerDe.java | 7 +-
.../common/testutils/HoodieTestDataGenerator.java | 215 ++++++-
.../hudi/common/testutils/HoodieTestUtils.java | 2 +-
.../reader/HoodieAvroRecordTestMerger.java | 89 ---
.../apache/hudi/common/util/TestBaseFileUtils.java | 27 +-
.../apache/hudi/common/util/TestConfigUtils.java | 13 +-
.../hudi/common/util/TestHoodieRecordUtils.java | 8 +-
.../schema/utils/TestAvroSchemaEvolutionUtils.java | 2 +-
.../org/apache/hudi/keygen/TestKeyGenerator.java | 2 +-
.../hudi/keygen/constant/TestKeyGeneratorType.java | 92 +++
.../hudi/metadata/TestHoodieIndexVersion.java | 6 +-
.../hudi/metadata/TestMetadataPartitionType.java | 37 +-
.../org/apache/hudi/stats/TestValueMetadata.java | 60 ++
.../java/org/apache/hudi/stats/TestValueType.java | 57 ++
hudi-examples/hudi-examples-flink/pom.xml | 4 +
.../examples/quickstart/HoodieSparkQuickstart.java | 1 -
hudi-flink-datasource/hudi-flink/pom.xml | 4 +
.../apache/hudi/configuration/OptionsResolver.java | 8 +
.../org/apache/hudi/sink/StreamWriteFunction.java | 3 +-
.../apache/hudi/sink/bulk/AutoRowDataKeyGen.java | 10 +-
.../org/apache/hudi/sink/bulk/RowDataKeyGen.java | 29 +-
.../hudi/table/format/cdc/CdcInputFormat.java | 10 +-
.../apache/hudi/sink/bulk/TestRowDataKeyGen.java | 34 +-
.../source/TestStreamReadMonitoringFunction.java | 4 +-
.../apache/hudi/table/ITTestSchemaEvolution.java | 8 +-
.../apache/hudi/utils/TestAvroSchemaConverter.java | 4 +-
.../common/table/ParquetTableSchemaResolver.java | 12 +-
.../java/org/apache/hudi/common/util/OrcUtils.java | 5 +-
.../org/apache/hudi/common/util/ParquetUtils.java | 76 ++-
.../org/apache/hudi/hadoop/fs/HadoopFSUtils.java | 4 +-
.../io/hadoop/HoodieAvroFileReaderFactory.java | 10 +
.../io/hadoop/HoodieAvroFileWriterFactory.java | 5 +-
.../avro/AvroSchemaConverterWithTimestampNTZ.java | 598 ++++++++++++++++++
.../avro/HoodieAvroParquetSchemaConverter.java | 57 ++
.../apache/parquet/avro/HoodieAvroReadSupport.java | 47 ++
.../parquet/avro/NativeAvroSchemaConverter.java} | 30 +-
.../apache/parquet/conf/ParquetConfiguration.java | 178 ++++++
.../org/apache/hudi/common/fs/TestFSUtils.java | 2 +-
.../common/functional/TestHoodieLogFormat.java | 4 +-
.../common/model/TestHoodieAvroIndexedRecord.java | 18 +-
.../hudi/common/table/TestHoodieTableConfig.java | 8 +-
.../common/table/TestHoodieTableMetaClient.java | 3 +-
.../common/table/read/TestCustomRecordMerger.java | 40 +-
.../TestHoodieFileGroupReaderInflightCommit.java | 4 +-
.../apache/hudi/common/util/TestParquetUtils.java | 5 +-
.../hudi/metadata/TestHoodieMetadataPayload.java | 24 +-
.../hudi/metadata/TestHoodieTableMetadataUtil.java | 66 +-
.../hudi/hadoop/DefaultHiveRecordMerger.java | 42 +-
.../org/apache/hudi/hadoop/HiveRecordContext.java | 14 +-
.../org/apache/hudi/hadoop/HoodieHiveRecord.java | 16 +-
.../OverwriteWithLatestHiveRecordMerger.java | 11 +-
.../apache/hudi/hadoop/SchemaEvolutionContext.java | 6 +
.../hudi/hadoop/avro/HoodieAvroParquetReader.java | 5 +-
.../realtime/AbstractRealtimeRecordReader.java | 3 +-
.../realtime/RealtimeCompactedRecordReader.java | 27 +-
.../utils/HoodieRealtimeRecordReaderUtils.java | 3 +-
.../reader/DFSHoodieDatasetInputReader.java | 6 +-
.../integ/testsuite/reader/SparkBasedReader.java | 9 +-
.../schema/TestSuiteFileBasedSchemaProvider.java | 4 +-
hudi-io/pom.xml | 5 +
.../java/org/apache/hudi/io/hfile/HFileBlock.java | 22 +-
.../org/apache/hudi/io/hfile/TestHFileWriter.java | 2 +-
.../org/apache/hudi/io/hfile/TestHfileBlock.java | 5 -
.../src/test/resources/hfile/hudi-generated.hfile | Bin 4654 -> 4674 bytes
hudi-spark-datasource/README.md | 6 +-
...va => HoodieVectorizedParquetRecordReader.java} | 5 +-
...org.apache.spark.sql.sources.DataSourceRegister | 2 +-
...DefaultSource.scala => BaseDefaultSource.scala} | 2 +-
.../org/apache/hudi/ColumnStatsIndexSupport.scala | 46 +-
.../scala/org/apache/hudi/DataSourceOptions.scala | 3 +
.../org/apache/hudi/ExpressionIndexSupport.scala | 38 +-
.../scala/org/apache/hudi/HoodieBaseRelation.scala | 4 +-
.../scala/org/apache/hudi/HoodieCDCFileIndex.scala | 2 +-
.../scala/org/apache/hudi/HoodieFileIndex.scala | 2 +-
.../org/apache/hudi/HoodieSparkSqlWriter.scala | 2 +
.../src/main/scala/org/apache/hudi/Iterators.scala | 55 +-
.../apache/hudi/PartitionDirectoryConverter.scala | 2 +-
.../apache/hudi/PartitionStatsIndexSupport.scala | 6 +-
.../org/apache/hudi/RecordLevelIndexSupport.scala | 5 +-
.../org/apache/hudi/SecondaryIndexSupport.scala | 4 +-
.../org/apache/hudi/SparkBaseIndexSupport.scala | 15 +-
.../apache/hudi/SparkHoodieTableFileIndex.scala | 5 +-
.../org/apache/hudi/cdc/CDCFileGroupIterator.scala | 4 +-
...ils.scala => BaseHoodieCatalystPlanUtils.scala} | 7 +-
.../sql/FileFormatUtilsForFileGroupReader.scala | 9 +-
.../spark/sql/HoodieSpark3CatalogUtils.scala | 46 --
.../spark/sql/HoodieSparkCatalogUtils.scala} | 17 +-
.../apache/spark/sql/ResolveInsertionBase.scala | 5 +-
.../spark/sql/adapter/BaseSpark3Adapter.scala | 114 ----
.../apache/spark/sql/avro/SchemaConverters.scala | 5 +-
.../sql/catalyst/catalog/HoodieCatalogTable.scala | 11 +-
.../HoodieFileSystemViewTableValuedFunction.scala | 4 +-
.../HoodieMetadataTableValuedFunction.scala | 4 +-
.../sql/catalyst/plans/logical/HoodieQuery.scala | 4 +-
.../plans/logical/HoodieTableChanges.scala | 6 +-
.../HoodieTimelineTableValuedFunction.scala | 4 +-
.../BaseHoodieNestedSchemaPruning.scala} | 46 +-
.../datasources/Spark3ParsePartitionUtil.scala | 283 ---------
.../HoodieFileGroupReaderBasedFileFormat.scala | 14 +-
...ils.scala => ParquetSchemaEvolutionUtils.scala} | 21 +-
.../parquet/SparkParquetReaderBase.scala | 2 +-
.../apache/spark/sql/hudi/DataSkippingUtils.scala | 14 +-
.../spark/sql/hudi/HoodieSqlCommonUtils.scala | 11 +-
.../spark/sql/hudi/ProvidesHoodieConfig.scala | 16 +-
...nalysis.scala => HoodieSparkBaseAnalysis.scala} | 15 +-
.../spark/sql/hudi/catalog/HoodieCatalog.scala | 7 +-
.../AlterHoodieTableAddColumnsCommand.scala | 7 +-
.../AlterHoodieTableAddPartitionCommand.scala | 9 +-
.../AlterHoodieTableChangeColumnCommand.scala | 9 +-
.../AlterHoodieTableDropPartitionCommand.scala | 5 +-
.../spark/sql/hudi/command/AlterTableCommand.scala | 2 +-
.../hudi/command/CreateHoodieTableCommand.scala | 7 +-
.../command/CreateHoodieTableLikeCommand.scala | 7 +-
.../sql/hudi/command/HoodieCommandMetrics.scala | 0
...odieSparkValidateDuplicateKeyRecordMerger.scala | 11 +-
.../hudi/command/RepairHoodieTableCommand.scala | 9 +-
.../hudi/command/TruncateHoodieTableCommand.scala | 7 +-
.../exception/HoodieAnalysisException.scala | 12 +-
.../hudi/command/payload/ExpressionPayload.scala | 8 +-
.../sql/hudi/streaming/HoodieStreamSourceV1.scala | 17 +-
.../sql/hudi/streaming/HoodieStreamSourceV2.scala | 18 +-
.../hudi/testutils/LogFileColStatsTestUtil.java | 2 +-
.../hudi/testutils/SparkDatasetTestUtils.java | 2 +-
...stSparkFileFormatInternalRowReaderContext.scala | 9 +-
.../datasources/orc/TestSparkOrcReaderFormat.scala | 2 +-
.../parquet/TestSparkParquetReaderFormat.scala | 2 +-
.../spark/sql/hudi/TestProvidesHoodieConfig.scala | 1 -
.../org/apache/spark/sql/hudi/SparkHelpers.scala | 4 +-
.../spark/sql/hudi/analysis/HoodieAnalysis.scala | 55 +-
.../command/procedures/HiveSyncProcedure.scala | 2 +-
.../procedures/HoodieProcedureFilterUtils.scala | 15 +
.../procedures/PartitionBucketIndexManager.scala | 9 +-
.../command/procedures/RunBootstrapProcedure.scala | 2 +-
.../procedures/TruncateTableProcedure.scala | 5 +-
.../spark/sql/parser/HoodieCommonSqlParser.scala | 6 +-
.../org/apache/hudi/ColumnStatsIndexHelper.java | 8 +-
.../java/org/apache/hudi/TestDataSourceUtils.java | 5 +-
.../apache/hudi/TestDefaultSparkRecordMerger.java | 93 ++-
.../hudi/TestHoodieMergeHandleWithSparkMerger.java | 22 +-
.../TestPositionBasedFileGroupRecordBuffer.java | 8 +-
.../TestMetadataUtilRLIandSIRecordGeneration.java | 12 +-
.../hudi/client/model/TestHoodieInternalRow.java | 81 ++-
.../TestBulkInsertInternalPartitioner.java | 49 +-
.../TestRDDSimpleBucketBulkInsertPartitioner.java | 0
.../hudi/functional/TestBufferedRecordMerger.java | 114 +++-
.../TestColStatsRecordWithMetadataRecord.java | 40 +-
.../functional/TestHiveTableSchemaEvolution.java | 2 +-
.../hudi/functional/TestHoodieBackedMetadata.java | 5 +-
.../TestHoodieDatasetBulkInsertHelper.java | 3 +-
.../hudi/functional/TestIncrementalQueries.java | 200 +++++++
.../functional/TestSparkSortAndSizeClustering.java | 19 +-
.../test/java/org/apache/hudi/io/CustomMerger.java | 62 ++
.../java/org/apache/hudi/io/CustomPayload.java | 79 +++
.../java/org/apache/hudi/io/TestMergeHandle.java | 128 +---
.../hudi/keygen/TestComplexKeyGenerator.java | 183 ------
...dieSparkMergeOnReadTableInsertUpdateDelete.java | 2 +-
.../hudi/table/upgrade/TestUpgradeDowngrade.java | 78 ++-
.../hudi/testutils/KeyGeneratorTestUtilities.java | 104 ----
.../test/resources/schema_with_logical_types.avsc | 5 +-
.../resources/upgrade-downgrade-fixtures/README.md | 21 +
.../generate-fixtures.sh | 103 ++--
.../mor-tables/hudi-v6-table-complex-keygen.zip | Bin 0 -> 67342 bytes
.../mor-tables/hudi-v8-table-complex-keygen.zip | Bin 0 -> 136082 bytes
.../mor-tables/hudi-v9-table-complex-keygen.zip | Bin 0 -> 137899 bytes
...la => generate-fixture-complex-keygen-v9.scala} | 32 +-
...scala => generate-fixture-complex-keygen.scala} | 39 +-
...ate-fixture.scala => generate-fixture-v9.scala} | 0
.../scala-templates/generate-fixture.scala | 7 +-
.../org/apache/hudi/TestDataSkippingUtils.scala | 13 +-
.../org/apache/hudi/TestDataSourceDefaults.scala | 127 +++-
.../apache/hudi/TestHoodieDataSourceHelper.scala | 17 +-
.../org/apache/hudi/TestHoodieSparkUtils.scala | 2 +-
.../hudi/TestPartitionDirectoryConverter.scala | 16 +-
.../org/apache/hudi/TestSparkFilterHelper.scala | 2 +-
.../model/TestHoodieRecordSerialization.scala | 7 +-
.../read/TestHoodieFileGroupReaderOnSpark.scala | 4 +-
.../hudi/functional/ColumnStatIndexTestBase.scala | 148 +++--
.../hudi/functional/RecordLevelIndexTestBase.scala | 2 +-
.../hudi/functional/TestBasicSchemaEvolution.scala | 10 +-
.../hudi/functional/TestBucketIndexSupport.scala | 20 +-
.../apache/hudi/functional/TestCOWDataSource.scala | 293 ++++++++-
.../hudi/functional/TestColumnStatsIndex.scala | 19 +-
.../functional/TestColumnStatsIndexWithSQL.scala | 9 +-
.../hudi/functional/TestEightToNineUpgrade.scala | 44 +-
.../TestIncrementalReadWithFullTableScan.scala | 22 +-
.../apache/hudi/functional/TestMORDataSource.scala | 270 ++++++++-
.../TestMetadataTableWithSparkDataSource.scala | 8 +-
.../functional/TestNestedPartitionPathSchema.scala | 147 +++++
.../functional/TestParquetColumnProjection.scala | 75 +--
.../TestPartitionBucketIndexSupport.scala | 54 +-
.../functional/TestPartitionStatsPruning.scala | 13 +-
.../TestPartitionedRecordLevelIndex.scala | 11 +-
.../functional/TestPayloadDeprecationFlow.scala | 265 +++++---
.../hudi/functional/TestRecordLevelIndex.scala | 6 +-
.../functional/TestRecordLevelIndexWithSQL.scala | 26 +-
.../functional/TestSecondaryIndexDataTypes.scala | 11 +-
.../functional/cdc/TestCDCStreamingSuite.scala | 24 +-
.../benchmark/CreateHandleBenchmark.scala | 167 ++++++
.../sql/hudi/common/HoodieSparkSqlTestBase.scala | 25 +-
.../sql/hudi/common/TestHoodieDataUtils.scala | 35 +-
.../hudi/common/TestHoodieInternalRowUtils.scala | 20 +-
.../spark/sql/hudi/ddl/TestCreateTable.scala | 242 +++++++-
.../sql/hudi/dml/insert/TestInsertTable.scala | 2 +-
.../sql/hudi/dml/others/TestMergeIntoTable.scala | 71 +++
.../sql/hudi/dml/others/TestMergeIntoTable2.scala | 12 +-
.../hudi/feature/index/TestExpressionIndex.scala | 99 +--
.../TestHoodieBackedTableMetadataIndexLookup.scala | 4 +-
.../sql/hudi/feature/index/TestIndexSyntax.scala | 2 +-
.../hudi/feature/index/TestSecondaryIndex.scala | 3 +-
.../sql/hudi/procedure/TestCallCommandParser.scala | 56 +-
.../sql/hudi/procedure/TestCallProcedure.scala | 1 +
.../hudi/procedure/TestSavepointsProcedure.scala | 1 +
.../hudi-spark3-common}/pom.xml | 211 +++----
.../hudi/client/model/Spark3HoodieInternalRow.java | 24 +-
...Spark3HoodiePartitionCDCFileGroupMapping.scala} | 9 +-
.../Spark3HoodiePartitionFileSliceMapping.scala} | 11 +-
.../apache/hudi/Spark3HoodiePartitionValues.scala} | 4 +-
.../org/apache/hudi/Spark3HoodieUTF8String.scala} | 11 +-
.../org/apache/spark/sql/Spark3DataFrameUtil.scala | 2 +-
.../spark/sql/Spark3HoodieUTF8StringFactory.scala | 14 +-
.../apache/spark/sql/Spark3HoodieUnsafeUtils.scala | 5 +-
.../spark/sql/adapter/BaseSpark3Adapter.scala | 184 ++++++
.../Spark3HoodiePruneFileSourcePartitions.scala} | 6 +-
.../command/CreateHoodieTableAsSelectCommand.scala | 8 +-
.../hudi/command/DeleteHoodieTableCommand.scala | 5 +-
.../command/InsertIntoHoodieTableCommand.scala | 2 +-
.../hudi/command/MergeIntoHoodieTableCommand.scala | 28 +-
.../hudi/command/UpdateHoodieTableCommand.scala | 11 +-
hudi-spark-datasource/hudi-spark3.3.x/pom.xml | 7 +
...org.apache.spark.sql.sources.DataSourceRegister | 19 -
.../spark/sql/HoodieSpark33CatalystPlanUtils.scala | 2 +-
.../spark/sql/HoodieSpark33SchemaUtils.scala | 12 +
.../apache/spark/sql/adapter/Spark3_3Adapter.scala | 5 +-
.../datasources/Spark33NestedSchemaPruning.scala | 179 +-----
.../Spark33LegacyHoodieParquetFileFormat.scala | 2 +-
.../datasources/parquet/Spark33ParquetReader.scala | 2 +-
.../parser/HoodieSpark3_3ExtendedSqlParser.scala | 2 +-
hudi-spark-datasource/hudi-spark3.4.x/pom.xml | 7 +
...org.apache.spark.sql.sources.DataSourceRegister | 19 -
.../spark/sql/HoodieSpark34CatalystPlanUtils.scala | 2 +-
.../spark/sql/HoodieSpark34SchemaUtils.scala | 12 +
.../apache/spark/sql/adapter/Spark3_4Adapter.scala | 2 -
.../datasources/Spark34NestedSchemaPruning.scala | 179 +-----
.../Spark34LegacyHoodieParquetFileFormat.scala | 2 +-
.../datasources/parquet/Spark34ParquetReader.scala | 2 +-
.../parser/HoodieSpark3_4ExtendedSqlParser.scala | 2 +-
hudi-spark-datasource/hudi-spark3.5.x/pom.xml | 7 +
...org.apache.spark.sql.sources.DataSourceRegister | 19 -
.../spark/sql/HoodieSpark35CatalystPlanUtils.scala | 2 +-
.../spark/sql/HoodieSpark35SchemaUtils.scala | 12 +
.../apache/spark/sql/adapter/Spark3_5Adapter.scala | 2 -
.../datasources/Spark35NestedSchemaPruning.scala | 178 +-----
.../Spark35LegacyHoodieParquetFileFormat.scala | 2 +-
.../datasources/parquet/Spark35ParquetReader.scala | 2 +-
.../parser/HoodieSpark3_5ExtendedSqlParser.scala | 2 +-
.../hudi-spark4-common}/pom.xml | 211 +++----
.../hudi/client/model/Spark4HoodieInternalRow.java | 47 ++
...Spark4HoodiePartitionCDCFileGroupMapping.scala} | 9 +-
.../Spark4HoodiePartitionFileSliceMapping.scala} | 11 +-
.../apache/hudi/Spark4HoodiePartitionValues.scala} | 10 +-
.../org/apache/hudi/Spark4HoodieUTF8String.scala} | 12 +-
.../spark/sql/ExpressionColumnNodeWrapper.scala | 12 +-
.../sql/HoodieSpark4CatalystExpressionUtils.scala} | 3 +-
.../org/apache/spark/sql/Spark4DataFrameUtil.scala | 6 +-
.../spark/sql/Spark4HoodieUTF8StringFactory.scala | 13 +-
.../apache/spark/sql/Spark4HoodieUnsafeUtils.scala | 15 +-
.../spark/sql/adapter/BaseSpark4Adapter.scala | 189 ++++++
.../Spark4HoodiePruneFileSourcePartitions.scala} | 8 +-
.../command/CreateHoodieTableAsSelectCommand.scala | 11 +-
.../hudi/command/DeleteHoodieTableCommand.scala | 10 +-
.../command/InsertIntoHoodieTableCommand.scala | 2 +-
.../hudi/command/MergeIntoHoodieTableCommand.scala | 36 +-
.../hudi/command/UpdateHoodieTableCommand.scala | 16 +-
.../{hudi-spark3.5.x => hudi-spark4.0.x}/pom.xml | 33 +-
.../src/main/antlr4/imports/SqlBase.g4 | 0
.../apache/hudi/spark/sql/parser/HoodieSqlBase.g4 | 0
.../apache/hudi/Spark40HoodieFileScanRDD.scala} | 2 +-
.../HoodieSpark40CatalystExpressionUtils.scala} | 2 +-
.../sql/HoodieSpark40CatalystPlanUtils.scala} | 6 +-
.../spark/sql/HoodieSpark40SchemaUtils.scala} | 14 +-
.../spark/sql/adapter/Spark4_0Adapter.scala} | 36 +-
.../apache/spark/sql/avro/AvroDeserializer.scala | 0
.../org/apache/spark/sql/avro/AvroSerializer.scala | 0
.../org/apache/spark/sql/avro/AvroUtils.scala | 0
.../sql/avro/HoodieSpark4_0AvroDeserializer.scala} | 2 +-
.../sql/avro/HoodieSpark4_0AvroSerializer.scala} | 2 +-
.../HoodieSpark40PartitionedFileUtils.scala} | 4 +-
.../datasources/Spark40NestedSchemaPruning.scala | 61 ++
.../datasources/orc/Spark40OrcReader.scala} | 8 +-
.../parquet/Spark40DataSourceUtils.scala} | 2 +-
.../Spark40LegacyHoodieParquetFileFormat.scala} | 48 +-
.../parquet/Spark40ParquetReader.scala} | 14 +-
.../Spark40ResolveHudiAlterTableCommand.scala} | 5 +-
.../sql/hudi/analysis/HoodieSpark40Analysis.scala} | 19 +-
.../HoodieSpark4_0ExtendedSqlAstBuilder.scala} | 16 +-
.../parser/HoodieSpark4_0ExtendedSqlParser.scala} | 19 +-
.../sync/datahub/config/DataHubSyncConfig.java | 72 ++-
.../config/TlsEnabledDataHubEmitterSupplier.java | 176 ++++++
.../sync/datahub/config/TestDataHubSyncConfig.java | 88 +++
.../TestTlsEnabledDataHubEmitterSupplier.java | 460 ++++++++++++++
.../src/test/resources/multi-ca-cert.pem | 60 ++
.../src/test/resources/test-ca-cert.pem | 21 +
.../src/test/resources/test-keystore.p12 | Bin 0 -> 2706 bytes
.../src/test/resources/test-truststore.p12 | Bin 0 -> 2706 bytes
hudi-tests-common/pom.xml | 4 +
.../utilities/HoodieMetadataTableValidator.java | 17 +-
.../hudi/utilities/HoodieSnapshotExporter.java | 2 +-
.../org/apache/hudi/utilities/UtilHelpers.java | 8 +-
.../utilities/schema/KafkaOffsetPostProcessor.java | 4 +-
.../DeleteSupportSchemaPostProcessor.java | 4 +-
.../DropColumnSchemaPostProcessor.java | 3 +-
.../add/AddPrimitiveColumnSchemaPostProcessor.java | 3 +-
.../hudi/utilities/sources/HoodieIncrSource.java | 6 +-
.../sources/SnapshotLoadQuerySplitter.java | 4 +-
.../apache/hudi/utilities/sources/SqlSource.java | 1 -
.../helpers/MercifulJsonToRowConverter.java | 95 +++
.../utilities/sources/helpers/QueryRunner.java | 4 +-
.../apache/hudi/utilities/streamer/StreamSync.java | 3 +
.../TestHoodieMetadataTableValidator.java | 27 +-
.../hudi/utilities/TestHoodieRepairTool.java | 2 +-
.../deltastreamer/HoodieDeltaStreamerTestBase.java | 2 +-
.../deltastreamer/TestHoodieDeltaStreamer.java | 258 +++++++-
.../deltastreamer/TestSourceFormatAdapter.java | 3 +-
.../TestHoodieMultiTableServicesMain.java | 2 +-
.../utilities/sources/TestHoodieIncrSource.java | 17 +-
.../helpers/TestMercifulJsonToRowConverter.java | 67 ++-
.../sources/helpers/TestSanitizationUtils.java | 2 +-
...hotQuerySplitterWithoutPartitionFilterImpl.java | 49 ++
.../testutils/ColStatsUpgradeTesting.java | 160 +++++
.../utilities/testutils/UtilitiesTestBase.java | 2 +-
.../testutils/sources/AbstractBaseTestSource.java | 14 +-
.../transform/TestSqlQueryBasedTransformer.java | 3 +-
.../src/test/resources/col-stats/README.md | 40 ++
.../col-stats/colstats-upgrade-test-v6.zip | Bin 0 -> 212182 bytes
.../col-stats/colstats-upgrade-test-v8.zip | Bin 0 -> 260900 bytes
.../{source.avsc => source-timestamp-millis.avsc} | 4 +-
packaging/bundle-validation/base/Dockerfile | 2 +-
...h => build_flink1200hive313spark400scala213.sh} | 9 +-
packaging/bundle-validation/ci_run.sh | 17 +-
.../docker_java17/docker_java17_test.sh | 17 +-
packaging/bundle-validation/run_docker_java17.sh | 10 +
packaging/bundle-validation/validate.sh | 35 +-
packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh | 3 +-
packaging/hudi-flink-bundle/pom.xml | 8 +
packaging/hudi-hive-sync-bundle/pom.xml | 2 +
packaging/hudi-integ-test-bundle/pom.xml | 13 +
packaging/hudi-spark-bundle/pom.xml | 7 +
packaging/hudi-utilities-bundle/pom.xml | 7 +
pom.xml | 77 ++-
rfc/rfc-101/rfc-101.md | 4 +-
scripts/pr_compliance.py | 315 ++++++----
scripts/release/validate_source_copyright.sh | 2 +-
594 files changed, 14519 insertions(+), 6132 deletions(-)
delete mode 100644 .github/ISSUE_TEMPLATE/SUPPORT_REQUEST.md
create mode 100644 .github/ISSUE_TEMPLATE/hudi_bug.yml
create mode 100644 .github/ISSUE_TEMPLATE/hudi_community_support.yml
create mode 100644 .github/ISSUE_TEMPLATE/hudi_feature.yml
create mode 100644 .github/ISSUE_TEMPLATE/hudi_improvement.yml
create mode 100644 .github/workflows/pr_title_validation.yml
copy docker/compose/{docker-compose_hadoop284_hive233_spark353_arm64.yml => docker-compose_hadoop334_hive313_spark353_arm64.yml} (61%)
rename {hudi-flink-datasource/hudi-flink => hudi-client/hudi-flink-client}/src/main/java/org/apache/hudi/table/format/FlinkRecordContext.java (97%)
rename {hudi-flink-datasource/hudi-flink => hudi-client/hudi-flink-client}/src/main/java/org/apache/hudi/util/RecordKeyToRowDataConverter.java (100%)
create mode 100644 hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/stats/SparkValueMetadataUtils.java
copy hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/util/TableUtils.java => hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodiePartitionCDCFileGroupMapping.scala (79%)
copy hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ProcedureArgs.scala => hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodiePartitionFileSliceMapping.scala (78%)
copy hudi-client/hudi-spark-client/src/main/scala/org/apache/{spark/sql/HoodieCatalogUtils.scala => hudi/HoodiePartitionValues.scala} (76%)
copy hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/package.scala => hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieUTF8String.scala (59%)
delete mode 100644 hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkJdbcUtils.scala
rename hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/Spark3RowSerDe.scala => hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkRowSerDe.scala (87%)
rename {hudi-spark-datasource/hudi-spark-common/src/main/java => hudi-client/hudi-spark-client/src/main/scala}/org/apache/hudi/spark/internal/ReflectUtil.java (100%)
rename hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/HoodieSpark35CatalogUtils.scala => hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieUTF8StringFactory.scala (63%)
create mode 100644 hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/KeyGeneratorTestUtilities.java
create mode 100644 hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestComplexKeyGenerator.java
rename {hudi-spark-datasource/hudi-spark => hudi-client/hudi-spark-client}/src/test/java/org/apache/hudi/keygen/TestCustomKeyGenerator.java (70%)
rename {hudi-spark-datasource/hudi-spark => hudi-client/hudi-spark-client}/src/test/java/org/apache/hudi/keygen/TestGlobalDeleteRecordGenerator.java (50%)
rename {hudi-spark-datasource/hudi-spark => hudi-client/hudi-spark-client}/src/test/java/org/apache/hudi/keygen/TestNonpartitionedKeyGenerator.java (54%)
rename {hudi-spark-datasource/hudi-spark => hudi-client/hudi-spark-client}/src/test/java/org/apache/hudi/keygen/TestSimpleKeyGenerator.java (79%)
rename {hudi-spark-datasource/hudi-spark => hudi-client/hudi-spark-client}/src/test/java/org/apache/hudi/keygen/TestTimestampBasedKeyGenerator.java (79%)
rename {hudi-spark-datasource/hudi-spark => hudi-client/hudi-spark-client}/src/test/java/org/apache/hudi/keygen/factory/TestCreateKeyGeneratorByTypeWithFactory.java (87%)
rename {hudi-spark-datasource/hudi-spark => hudi-client/hudi-spark-client}/src/test/java/org/apache/hudi/keygen/factory/TestHoodieSparkKeyGeneratorFactory.java (91%)
rename hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/HoodieSpark33CatalogUtils.scala => hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/DisableOnSpark4Condition.java (56%)
copy hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ProcedureArgs.scala => hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/DisabledOnSpark4.java (65%)
copy packaging/hudi-gcp-bundle/src/main/java/org/apache/hudi/gcp/bigquery/bundle/Main.java => hudi-common/src/main/java/org/apache/hudi/ParquetAdapter.java (52%)
create mode 100644 hudi-common/src/main/java/org/apache/hudi/avro/JoinedGenericRecord.java
delete mode 100644 hudi-common/src/main/java/org/apache/hudi/common/model/HoodieMetadataRecordMerger.java
copy hudi-common/src/main/java/org/apache/hudi/{common/util/MapUtils.java => index/record/HoodieRecordIndex.java} (56%)
rename hudi-common/src/main/java/org/apache/hudi/{common/model => stats}/HoodieColumnRangeMetadata.java (69%)
create mode 100644 hudi-common/src/main/java/org/apache/hudi/stats/ValueMetadata.java
create mode 100644 hudi-common/src/main/java/org/apache/hudi/stats/ValueType.java
create mode 100644 hudi-common/src/main/java/org/apache/parquet/schema/LogicalTypeParquetAdapter.java
create mode 100644 hudi-common/src/main/java/org/apache/parquet/schema/OriginalTypeParquetAdapter.java
delete mode 100644 hudi-common/src/test/java/org/apache/hudi/common/model/TestHoodieMetadataRecordMerger.java
delete mode 100644 hudi-common/src/test/java/org/apache/hudi/common/testutils/reader/HoodieAvroRecordTestMerger.java
create mode 100644 hudi-common/src/test/java/org/apache/hudi/keygen/constant/TestKeyGeneratorType.java
create mode 100644 hudi-common/src/test/java/org/apache/hudi/stats/TestValueMetadata.java
create mode 100644 hudi-common/src/test/java/org/apache/hudi/stats/TestValueType.java
create mode 100644 hudi-hadoop-common/src/main/java/org/apache/parquet/avro/AvroSchemaConverterWithTimestampNTZ.java
create mode 100644 hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroParquetSchemaConverter.java
copy hudi-hadoop-common/src/{test/java/org/apache/hudi/storage/hadoop/TestHadoopStorageConfiguration.java => main/java/org/apache/parquet/avro/NativeAvroSchemaConverter.java} (58%)
create mode 100644 hudi-hadoop-common/src/main/java/org/apache/parquet/conf/ParquetConfiguration.java
rename hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/spark/sql/execution/datasources/parquet/{Spark3HoodieVectorizedParquetRecordReader.java => HoodieVectorizedParquetRecordReader.java} (97%)
rename hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/{Spark3DefaultSource.scala => BaseDefaultSource.scala} (94%)
rename hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/{HoodieSpark3CatalystPlanUtils.scala => BaseHoodieCatalystPlanUtils.scala} (94%)
delete mode 100644 hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalogUtils.scala
rename hudi-spark-datasource/{hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/HoodieSpark34CatalogUtils.scala => hudi-spark-common/src/main/scala/org/apache/spark/sql/HoodieSparkCatalogUtils.scala} (69%)
delete mode 100644 hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark3Adapter.scala
copy hudi-spark-datasource/{hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/execution/datasources/Spark34NestedSchemaPruning.scala => hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/BaseHoodieNestedSchemaPruning.scala} (80%)
delete mode 100644 hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/Spark3ParsePartitionUtil.scala
rename hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/{Spark3ParquetSchemaEvolutionUtils.scala => ParquetSchemaEvolutionUtils.scala} (93%)
rename hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/analysis/{HoodieSpark3Analysis.scala => HoodieSparkBaseAnalysis.scala} (96%)
rename hudi-spark-datasource/{hudi-spark => hudi-spark-common}/src/main/scala/org/apache/spark/sql/hudi/command/HoodieCommandMetrics.scala (100%)
copy hudi-aws/src/main/java/org/apache/hudi/aws/sync/util/GluePartitionFilterGenerator.java => hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/exception/HoodieAnalysisException.scala (70%)
rename hudi-spark-datasource/{hudi-spark => hudi-spark-common}/src/main/scala/org/apache/spark/sql/hudi/command/payload/ExpressionPayload.scala (98%)
rename {hudi-client/hudi-spark-client => hudi-spark-datasource/hudi-spark}/src/test/java/org/apache/hudi/client/model/TestHoodieInternalRow.java (80%)
rename {hudi-client/hudi-spark-client => hudi-spark-datasource/hudi-spark}/src/test/java/org/apache/hudi/execution/bulkinsert/TestBulkInsertInternalPartitioner.java (86%)
rename {hudi-client/hudi-spark-client => hudi-spark-datasource/hudi-spark}/src/test/java/org/apache/hudi/execution/bulkinsert/TestRDDSimpleBucketBulkInsertPartitioner.java (100%)
create mode 100644 hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestIncrementalQueries.java
create mode 100644 hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/CustomMerger.java
create mode 100644 hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/CustomPayload.java
delete mode 100644 hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/keygen/TestComplexKeyGenerator.java
delete mode 100644 hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/testutils/KeyGeneratorTestUtilities.java
create mode 100644 hudi-spark-datasource/hudi-spark/src/test/resources/upgrade-downgrade-fixtures/mor-tables/hudi-v6-table-complex-keygen.zip
create mode 100644 hudi-spark-datasource/hudi-spark/src/test/resources/upgrade-downgrade-fixtures/mor-tables/hudi-v8-table-complex-keygen.zip
create mode 100644 hudi-spark-datasource/hudi-spark/src/test/resources/upgrade-downgrade-fixtures/mor-tables/hudi-v9-table-complex-keygen.zip
copy hudi-spark-datasource/hudi-spark/src/test/resources/upgrade-downgrade-fixtures/scala-templates/{generate-fixture.scala => generate-fixture-complex-keygen-v9.scala} (79%)
copy hudi-spark-datasource/hudi-spark/src/test/resources/upgrade-downgrade-fixtures/scala-templates/{generate-fixture.scala => generate-fixture-complex-keygen.scala} (76%)
copy hudi-spark-datasource/hudi-spark/src/test/resources/upgrade-downgrade-fixtures/scala-templates/{generate-fixture.scala => generate-fixture-v9.scala} (100%)
create mode 100644 hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestNestedPartitionPathSchema.scala
create mode 100644 hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/CreateHandleBenchmark.scala
copy {hudi-examples/hudi-examples-spark => hudi-spark-datasource/hudi-spark3-common}/pom.xml (64%)
rename hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkRowSerDe.java => hudi-spark-datasource/hudi-spark3-common/src/main/java/org/apache/hudi/client/model/Spark3HoodieInternalRow.java (54%)
copy hudi-spark-datasource/{hudi-spark-common/src/main/scala/org/apache/hudi/HoodiePartitionCDCFileGroupMapping.scala => hudi-spark3-common/src/main/scala/org/apache/hudi/Spark3HoodiePartitionCDCFileGroupMapping.scala} (74%)
copy hudi-spark-datasource/{hudi-spark-common/src/main/scala/org/apache/hudi/HoodiePartitionFileSliceMapping.scala => hudi-spark3-common/src/main/scala/org/apache/hudi/Spark3HoodiePartitionFileSliceMapping.scala} (73%)
copy hudi-spark-datasource/{hudi-spark-common/src/main/scala/org/apache/hudi/HoodiePartitionValues.scala => hudi-spark3-common/src/main/scala/org/apache/hudi/Spark3HoodiePartitionValues.scala} (95%)
copy hudi-spark-datasource/{hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ProcedureArgs.scala => hudi-spark3-common/src/main/scala/org/apache/hudi/Spark3HoodieUTF8String.scala} (75%)
copy hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/DataFrameUtil.scala => hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/Spark3DataFrameUtil.scala (96%)
copy hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalogUtils.scala => hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/Spark3HoodieUTF8StringFactory.scala (73%)
copy hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieUnsafeUtils.scala => hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/Spark3HoodieUnsafeUtils.scala (98%)
create mode 100644 hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark3Adapter.scala
copy hudi-spark-datasource/{hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodiePruneFileSourcePartitions.scala => hudi-spark3-common/src/main/scala/org/apache/spark/sql/hudi/analysis/Spark3HoodiePruneFileSourcePartitions.scala} (93%)
copy hudi-spark-datasource/{hudi-spark => hudi-spark3-common}/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableAsSelectCommand.scala (95%)
copy hudi-spark-datasource/{hudi-spark => hudi-spark3-common}/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala (95%)
copy hudi-spark-datasource/{hudi-spark => hudi-spark3-common}/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala (99%)
copy hudi-spark-datasource/{hudi-spark => hudi-spark3-common}/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala (97%)
copy hudi-spark-datasource/{hudi-spark => hudi-spark3-common}/src/main/scala/org/apache/spark/sql/hudi/command/UpdateHoodieTableCommand.scala (92%)
delete mode 100644 hudi-spark-datasource/hudi-spark3.3.x/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
delete mode 100644 hudi-spark-datasource/hudi-spark3.4.x/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
delete mode 100644 hudi-spark-datasource/hudi-spark3.5.x/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
copy {hudi-examples/hudi-examples-spark => hudi-spark-datasource/hudi-spark4-common}/pom.xml (64%)
create mode 100644 hudi-spark-datasource/hudi-spark4-common/src/main/java/org/apache/hudi/client/model/Spark4HoodieInternalRow.java
rename hudi-spark-datasource/{hudi-spark-common/src/main/scala/org/apache/hudi/HoodiePartitionCDCFileGroupMapping.scala => hudi-spark4-common/src/main/scala/org/apache/hudi/Spark4HoodiePartitionCDCFileGroupMapping.scala} (74%)
rename hudi-spark-datasource/{hudi-spark-common/src/main/scala/org/apache/hudi/HoodiePartitionFileSliceMapping.scala => hudi-spark4-common/src/main/scala/org/apache/hudi/Spark4HoodiePartitionFileSliceMapping.scala} (73%)
rename hudi-spark-datasource/{hudi-spark-common/src/main/scala/org/apache/hudi/HoodiePartitionValues.scala => hudi-spark4-common/src/main/scala/org/apache/hudi/Spark4HoodiePartitionValues.scala} (91%)
copy hudi-spark-datasource/{hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ProcedureArgs.scala => hudi-spark4-common/src/main/scala/org/apache/hudi/Spark4HoodieUTF8String.scala} (69%)
copy hudi-common/src/main/java/org/apache/hudi/common/HoodieSchemaNotFoundException.java => hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/ExpressionColumnNodeWrapper.scala (72%)
copy hudi-spark-datasource/{hudi-spark-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystExpressionUtils.scala => hudi-spark4-common/src/main/scala/org/apache/spark/sql/HoodieSpark4CatalystExpressionUtils.scala} (96%)
copy hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/DataFrameUtil.scala => hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/Spark4DataFrameUtil.scala (83%)
rename hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalogUtils.scala => hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/Spark4HoodieUTF8StringFactory.scala (73%)
copy hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieUnsafeUtils.scala => hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/Spark4HoodieUnsafeUtils.scala (87%)
create mode 100644 hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark4Adapter.scala
rename hudi-spark-datasource/{hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodiePruneFileSourcePartitions.scala => hudi-spark4-common/src/main/scala/org/apache/spark/sql/hudi/analysis/Spark4HoodiePruneFileSourcePartitions.scala} (92%)
rename hudi-spark-datasource/{hudi-spark => hudi-spark4-common}/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableAsSelectCommand.scala (93%)
rename hudi-spark-datasource/{hudi-spark => hudi-spark4-common}/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala (90%)
rename hudi-spark-datasource/{hudi-spark => hudi-spark4-common}/src/main/scala/org/apache/spark/sql/hudi/command/InsertIntoHoodieTableCommand.scala (99%)
rename hudi-spark-datasource/{hudi-spark => hudi-spark4-common}/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala (96%)
rename hudi-spark-datasource/{hudi-spark => hudi-spark4-common}/src/main/scala/org/apache/spark/sql/hudi/command/UpdateHoodieTableCommand.scala (90%)
copy hudi-spark-datasource/{hudi-spark3.5.x => hudi-spark4.0.x}/pom.xml (92%)
copy hudi-spark-datasource/{hudi-spark3.3.x => hudi-spark4.0.x}/src/main/antlr4/imports/SqlBase.g4 (100%)
copy hudi-spark-datasource/{hudi-spark3.3.x => hudi-spark4.0.x}/src/main/antlr4/org/apache/hudi/spark/sql/parser/HoodieSqlBase.g4 (100%)
copy hudi-spark-datasource/{hudi-spark3.4.x/src/main/scala/org/apache/hudi/Spark34HoodieFileScanRDD.scala => hudi-spark4.0.x/src/main/scala/org/apache/hudi/Spark40HoodieFileScanRDD.scala} (96%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/HoodieSpark35CatalystExpressionUtils.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/HoodieSpark40CatalystExpressionUtils.scala} (98%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/HoodieSpark35CatalystPlanUtils.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/HoodieSpark40CatalystPlanUtils.scala} (97%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/HoodieSpark35SchemaUtils.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/HoodieSpark40SchemaUtils.scala} (73%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/adapter/Spark3_5Adapter.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/adapter/Spark4_0Adapter.scala} (87%)
copy hudi-spark-datasource/{hudi-spark3.5.x => hudi-spark4.0.x}/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala (100%)
copy hudi-spark-datasource/{hudi-spark3.5.x => hudi-spark4.0.x}/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala (100%)
copy hudi-spark-datasource/{hudi-spark3.4.x => hudi-spark4.0.x}/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala (100%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_5AvroDeserializer.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark4_0AvroDeserializer.scala} (95%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark3_5AvroSerializer.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/avro/HoodieSpark4_0AvroSerializer.scala} (94%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/execution/datasources/HoodieSpark35PartitionedFileUtils.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/HoodieSpark40PartitionedFileUtils.scala} (96%)
create mode 100644 hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/Spark40NestedSchemaPruning.scala
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/execution/datasources/orc/Spark35OrcReader.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/orc/Spark40OrcReader.scala} (95%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark35DataSourceUtils.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark40DataSourceUtils.scala} (99%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark35LegacyHoodieParquetFileFormat.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark40LegacyHoodieParquetFileFormat.scala} (93%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark35ParquetReader.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark40ParquetReader.scala} (96%)
copy hudi-spark-datasource/{hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/hudi/Spark34ResolveHudiAlterTableCommand.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/hudi/Spark40ResolveHudiAlterTableCommand.scala} (93%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark35Analysis.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark40Analysis.scala} (96%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_5ExtendedSqlAstBuilder.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_0ExtendedSqlAstBuilder.scala} (99%)
copy hudi-spark-datasource/{hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_5ExtendedSqlParser.scala => hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_0ExtendedSqlParser.scala} (92%)
create mode 100644 hudi-sync/hudi-datahub-sync/src/main/java/org/apache/hudi/sync/datahub/config/TlsEnabledDataHubEmitterSupplier.java
create mode 100644 hudi-sync/hudi-datahub-sync/src/test/java/org/apache/hudi/sync/datahub/config/TestTlsEnabledDataHubEmitterSupplier.java
create mode 100644 hudi-sync/hudi-datahub-sync/src/test/resources/multi-ca-cert.pem
create mode 100644 hudi-sync/hudi-datahub-sync/src/test/resources/test-ca-cert.pem
create mode 100644 hudi-sync/hudi-datahub-sync/src/test/resources/test-keystore.p12
create mode 100644 hudi-sync/hudi-datahub-sync/src/test/resources/test-truststore.p12
create mode 100644 hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestSnapshotQuerySplitterWithoutPartitionFilterImpl.java
create mode 100644 hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/ColStatsUpgradeTesting.java
create mode 100644 hudi-utilities/src/test/resources/col-stats/README.md
create mode 100644 hudi-utilities/src/test/resources/col-stats/colstats-upgrade-test-v6.zip
create mode 100644 hudi-utilities/src/test/resources/col-stats/colstats-upgrade-test-v8.zip
copy hudi-utilities/src/test/resources/streamer-config/{source.avsc => source-timestamp-millis.avsc} (98%)
copy packaging/bundle-validation/base/{build_flink1200hive313spark351scala213.sh => build_flink1200hive313spark400scala213.sh} (79%)