You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by gi...@apache.org on 2024/07/07 06:10:57 UTC
(iceberg) branch dependabot/gradle/software.amazon.s3.accessgrants-aws-s3-accessgrants-java-plugin-2.1.0 updated (1270559645 -> 003cd06065)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/gradle/software.amazon.s3.accessgrants-aws-s3-accessgrants-java-plugin-2.1.0
in repository https://gitbox.apache.org/repos/asf/iceberg.git
discard 1270559645 Build: Bump software.amazon.s3.accessgrants:aws-s3-accessgrants-java-plugin
add fa95e1277f Build: Bump mkdocs-material from 9.5.25 to 9.5.26 (#10464)
add 282fa73b2c Pin 3rd party CI action version (#10481)
add bdd6225b62 Build: Bump io.delta:delta-standalone_2.12 from 3.1.0 to 3.2.0 (#10321)
add b6c949cd86 Core, Spark: Calling rewrite_position_delete_files fails on tables with more than 1k columns (#10020)
add 74cf6977b9 Build: Bump Nessie to 0.90.4 (#10492)
add a02b551825 Core: Simplify `loadCatalog` method call in Iceberg (#10488)
add 42c3159226 MR: Optimize schema string retrieval in Iceberg (#10489)
add b7a0bea6e5 Build: Rename allVersions flag to allModules (#10499)
add 636f963049 Build: Bump net.snowflake:snowflake-jdbc from 3.16.0 to 3.16.1 (#10419)
add 6c76a3a9a7 Build: Bump org.scala-lang.modules:scala-collection-compat_2.13 (#10195)
add 31654239eb Build: Bump org.springframework:spring-web from 5.3.36 to 5.3.37 (#10503)
add 26c298a934 Build: Bump io.netty:netty-buffer from 4.1.110.Final to 4.1.111.Final (#10504)
add 26dc338a4d Build: Bump software.amazon.awssdk:bom from 2.25.69 to 2.26.3 (#10505)
add a3a2b585b9 Build: Bump datamodel-code-generator from 0.25.6 to 0.25.7 (#10507)
add 3d3e565a79 Core: Remove deprecated APIs scheduled for removal in 1.6.0 (#10501)
add 5f3809c515 Core, Flink, Spark: Import the right assertThatThrownBy method from AssertJ (#10512)
add 87810c8f2b AWS: Rename test helper to deconflict with Assertions (#10511)
add 1928257ce3 Build: Merge job definitions in spark-ci.yml (#10513)
add 52d82f93ef API, Spark 3.3: Remove all usages of deprecated AssertHelpers (#10500)
add d8f26caa97 Spark: Use bulk deletes in rewrite manifests action (#10343)
add 4859b552e9 Build: Update NOTICE to include copyright for 2024 (#10471)
add ae6f9066ee Remove redundant `-XX:+IgnoreUnrecognizedVMOptions` (#10475)
add 2289758dc9 spec: Fix formatting of Default values (#10525)
add 2862047771 docs: Introduce variable for setting the Flink version (#10463)
add 0b505de413 Flink: Import Assertions statically (#10532)
add 316f0a11bf Spark: Import Assertions statically (#10531)
add 5ea78e3fbe Statically import methods from AssertJ Assertions (#10517)
add 3c71440277 Core: Simplify newTableMetadata method in TableMetadata (#10528)
add 08cc776bfe Build: Run revapi workflow on workflow/build system changes (#10485)
add 9dbfbbb203 API, Core, Flink, Avro, Parquet: Remove dead code and update javadocs (#10530)
add 791140652c Fix JVM locale dependent casing (#10521)
add 8248663a2a Fix code depending on JVM default charset (#10529)
add cbd11d7aa5 Run Flink tests on Java 17 (#10477)
add 10d7ab11c9 Run Hive3 tests on Java 11 and 17 too (#10482)
add 23a578e5c6 Core: Prevent duplicate data/delete files (#10007)
add 601efaf309 Flink: Fix the condition of `formatVersion` for skipping test cases (#10541)
add a2a679f7c0 Build: Sort error-prone configuration options (#10540)
add c67c9124d3 Core, Spark: Spark writes/actions should only perform cleanup if failure is cleanable (#10373)
add 1ec69d1ce9 Docs: Allow Java 17 in contribute.md (#10545)
add e57b9f6954 Spark: Backport #10373 to Spark 3.3 and 3.4 (#10546)
add a47937c0c1 Spark 3.5: Support Aggregate push down for incremental scan (#10538)
add da6268ef19 Build: Bump mkdocs-material from 9.5.26 to 9.5.27 (#10555)
add 87d2fdd173 Build: Bump software.amazon.awssdk:bom from 2.26.3 to 2.26.7 (#10554)
add a75fb1c961 Build: Bump nessie from 0.90.4 to 0.91.1 (#10551)
add 29fd2a0cb9 Build: Bump org.roaringbitmap:RoaringBitmap from 1.0.6 to 1.1.0 (#10552)
add 5733aecd0d Core: Pushdown data_file.content filter in entries metadata table (#10203)
add c85f66759e Build: Bump Nessie to 0.91.2 (#10563)
add 4bd9e64f28 Spark: Remove useless code in `TestRemoveOrphanFilesProcedure` (#10562)
add 565352d12b Build: Upgrade to gradle 8.8 (#8486)
add 8af8a4da5e Spark: Backport support for Aggregate push down for incremental scan to Spark 3.4 (#10561)
add 10fc04b639 Build: Move to `goooler` shadow plugin (#10568)
add 29ff08a086 Spec: Fix Typo (#10564)
add 9ed33839e7 Core, Flink: Add task-type field to JSON serde of scan task / Add JSON serde for StaticDataTask. (#9728)
add c88e9422bb Azure: Make AzureProperties w/ shared-key creds serializable (#10045)
add 87a988fdda Spark 3.5: Parallelize reading files in snapshot and migrate procedures (#10037)
add f9f30f6b90 Docs: Add BigQuery docs url to sidebar (#10574)
add 91fbcaa62c Build: Run CI checks on all supported JDKs (#10473)
add c469edf6cd Common: DynConstructors cleanup (#10542)
add 7071dc18ed Fix CI script inclusion of release branches (#10514)
add 2bdae5e744 Fix incorrect double-checked-locking around TestStreamScanSql#tEnv (#10605)
add f0cb1ec2c6 Build: Bump nessie from 0.91.2 to 0.91.3 (#10608)
add 0e7aa84b1d Build: Bump software.amazon.awssdk:bom from 2.26.7 to 2.26.12 (#10611)
add a975a95552 Flink: Migrate HadoopCatalog related tests (#10358)
add d3cb1b696a Core: Fix ParallelIterable memory leak where queue continues to be populated even after iterator close (#9402)
add f4ddaea56b Core: Handle potential NPE in RESTSessionCatalog#newSessionCache (#10607)
add 3df3d29a69 Build: Bump io.github.goooler.shadow:shadow-gradle-plugin (#10612)
add 0eab91aad7 Flink 1.17, 1.18: Migrate HadoopCatalog related tests (#10620)
add afda8be256 Address IntelliJ inspection findings (#10583)
add d255c87b00 Build: Enable the Gradle build cache (#10602)
add 3825477b01 API, Flink, ORC: Fix implicit `long` casting issues (#10580)
add d1f9f28f4f REST: disallow overriding "credential" in table sessions (#10345)
add c7eba348d8 Apply IntelliJ inspection findings to older Spark + Flink versions (#10625)
add 4048987d2f Core: Assume issued_token_type is access_token to fully comply with RFC 6749 (#10314)
add 4aee307617 Flink: Fix `long` casting issues (#10629)
add fc213cc393 Core: Handle possible heap data corruption in OAuth2Util.AuthSession#headers (#10615)
add 9a0fc49481 AWS: Retain Glue Catalog column comment (#10276)
add 83dd59a6f8 Build: Use official revapi Gradle plugin (#10631)
add 24d26b6a35 Core: Fix create v1 table on REST Catalog (#10369)
add ce1279e3d0 Flink 1.19: Migrate source package to JUnit5 (#10632)
add 6223708dd5 Spark 3.5: Support read of partition metadata column when table has over 1k columns (#10547)
add 1c3476d953 Build: Bump com.azure:azure-sdk-bom from 1.2.24 to 1.2.25 (#10652)
add 003cd06065 Build: Bump software.amazon.s3.accessgrants:aws-s3-accessgrants-java-plugin
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (1270559645)
\
N -- N -- N refs/heads/dependabot/gradle/software.amazon.s3.accessgrants-aws-s3-accessgrants-java-plugin-2.1.0 (003cd06065)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.baseline/checkstyle/checkstyle.xml | 8 +
.github/workflows/api-binary-compatibility.yml | 7 +-
.github/workflows/delta-conversion-ci.yml | 4 +-
.github/workflows/flink-ci.yml | 10 +-
.github/workflows/hive-ci.yml | 9 +-
.github/workflows/java-ci.yml | 16 +-
.github/workflows/open-api.yml | 4 +-
.github/workflows/publish-snapshot.yml | 2 +-
.github/workflows/spark-ci.yml | 84 +-
.palantir/revapi.yml | 38 +
LICENSE | 1 -
NOTICE | 2 +-
README.md | 2 +-
.../org/apache/iceberg/aliyun/oss/BaseOSSFile.java | 2 +-
.../java/org/apache/iceberg/aliyun/oss/OSSURI.java | 3 +-
.../iceberg/aliyun/TestAliyunClientFactories.java | 19 +-
.../apache/iceberg/aliyun/oss/TestOSSFileIO.java | 47 +-
.../iceberg/aliyun/oss/TestOSSInputFile.java | 21 +-
.../iceberg/aliyun/oss/TestOSSInputStream.java | 14 +-
.../iceberg/aliyun/oss/TestOSSOutputFile.java | 35 +-
.../iceberg/aliyun/oss/TestOSSOutputStream.java | 15 +-
.../org/apache/iceberg/aliyun/oss/TestOSSURI.java | 41 +-
.../aliyun/oss/mock/AliyunOSSMockExtension.java | 2 +-
.../oss/mock/AliyunOSSMockLocalController.java | 4 +-
.../aliyun/oss/mock/TestLocalAliyunOSS.java | 24 +-
.../java/org/apache/iceberg/PartitionSpec.java | 31 +-
api/src/main/java/org/apache/iceberg/Schema.java | 70 +-
.../java/org/apache/iceberg/SortOrderBuilder.java | 2 +-
.../org/apache/iceberg/actions/MigrateTable.java | 12 +
.../org/apache/iceberg/actions/SnapshotTable.java | 12 +
.../apache/iceberg/catalog/TableIdentifier.java | 3 +-
.../apache/iceberg/expressions/ExpressionUtil.java | 2 +-
.../apache/iceberg/expressions/Projections.java | 4 -
.../main/java/org/apache/iceberg/io/FileIO.java | 2 +-
.../org/apache/iceberg/metrics/DefaultTimer.java | 15 +-
.../types/{AssignFreshIds.java => AssignIds.java} | 54 +-
.../java/org/apache/iceberg/types/TypeUtil.java | 16 +
.../main/java/org/apache/iceberg/types/Types.java | 4 +
.../java/org/apache/iceberg/util/BucketUtil.java | 4 +-
.../java/org/apache/iceberg/AssertHelpers.java | 213 -----
.../test/java/org/apache/iceberg/TestHelpers.java | 9 +-
.../iceberg/TestPartitionSpecValidation.java | 89 +-
.../java/org/apache/iceberg/TestSnapshotRef.java | 17 +-
.../apache/iceberg/TestSortOrderComparators.java | 48 +-
.../org/apache/iceberg/catalog/TestNamespace.java | 22 +-
.../iceberg/catalog/TestTableIdentifier.java | 12 +-
.../iceberg/expressions/TestAggregateBinding.java | 10 +-
.../apache/iceberg/expressions/TestEvaluator.java | 33 +-
.../iceberg/expressions/TestExpressionBinding.java | 10 +-
.../iceberg/expressions/TestExpressionHelpers.java | 8 +-
.../iceberg/expressions/TestExpressionUtil.java | 9 +-
.../TestInclusiveManifestEvaluator.java | 6 +-
.../expressions/TestInclusiveMetricsEvaluator.java | 6 +-
.../iceberg/expressions/TestPredicateBinding.java | 10 +-
.../expressions/TestStrictMetricsEvaluator.java | 5 +-
.../expressions/TestStringLiteralConversions.java | 6 +-
.../org/apache/iceberg/io/TestCloseableGroup.java | 13 +-
.../apache/iceberg/io/TestCloseableIterable.java | 34 +-
.../org/apache/iceberg/io/TestClosingIterator.java | 5 +-
.../iceberg/io/TestableCloseableIterable.java | 2 +-
.../apache/iceberg/metrics/TestDefaultCounter.java | 18 +-
.../iceberg/metrics/TestDefaultMetricsContext.java | 47 +-
.../apache/iceberg/metrics/TestDefaultTimer.java | 101 +-
.../metrics/TestFixedReservoirHistogram.java | 69 +-
.../apache/iceberg/transforms/TestBucketing.java | 6 +-
.../transforms/TestBucketingProjection.java | 9 +-
.../apache/iceberg/transforms/TestProjection.java | 12 +-
.../apache/iceberg/transforms/TestResiduals.java | 4 +-
.../apache/iceberg/transforms/TestStartsWith.java | 3 +-
.../apache/iceberg/transforms/TestTruncate.java | 4 +-
.../transforms/TestTruncatesProjection.java | 15 +-
.../org/apache/iceberg/types/TestTypeUtil.java | 17 +-
.../java/org/apache/iceberg/types/TestTypes.java | 22 +-
.../apache/iceberg/util/TestCharSequenceMap.java | 5 +-
.../apache/iceberg/util/TestCharSequenceSet.java | 41 +-
.../org/apache/iceberg/util/TestDateTimeUtil.java | 12 +-
.../org/apache/iceberg/util/TestExceptionUtil.java | 11 +-
.../GenericArrowVectorAccessorFactory.java | 6 -
.../parquet/BaseVectorizedParquetValuesReader.java | 4 +-
.../iceberg/arrow/vectorized/ArrowReaderTest.java | 5 +-
.../vectorized/parquet/DecimalVectorUtilTest.java | 4 +-
aws-bundle/NOTICE | 2 +-
aws-bundle/build.gradle | 2 +-
.../org/apache/iceberg/aws/glue/GlueTestBase.java | 36 +
.../iceberg/aws/glue/TestGlueCatalogTable.java | 129 +++
.../aws/lakeformation/LakeFormationTestBase.java | 15 +-
.../TestLakeFormationAwsClientFactory.java | 4 +-
.../iceberg/aws/s3/TestS3FileIOIntegration.java | 2 +-
.../iceberg/aws/s3/TestS3MultipartUpload.java | 2 +-
.../apache/iceberg/aws/AwsClientProperties.java | 2 +-
.../java/org/apache/iceberg/aws/AwsProperties.java | 18 +-
.../org/apache/iceberg/aws/glue/GlueCatalog.java | 2 +-
.../iceberg/aws/glue/GlueTableOperations.java | 6 +-
.../iceberg/aws/glue/IcebergToGlueConverter.java | 76 +-
.../LakeFormationAwsClientFactory.java | 4 +-
.../apache/iceberg/aws/s3/S3FileIOProperties.java | 16 +-
.../iceberg/aws/AwsClientPropertiesTest.java | 21 +-
.../iceberg/aws/HttpClientPropertiesTest.java | 11 +-
.../apache/iceberg/aws/TestAwsClientFactories.java | 56 +-
.../org/apache/iceberg/aws/TestAwsProperties.java | 6 +-
.../iceberg/aws/TestHttpClientProperties.java | 11 +-
.../apache/iceberg/aws/TestRESTSigV4Signer.java | 7 +-
.../aws/TestS3FileIOAwsClientFactories.java | 7 +-
.../apache/iceberg/aws/TestS3FileIOProperties.java | 63 +-
.../iceberg/aws/dynamodb/TestDynamoDbCatalog.java | 11 +-
.../apache/iceberg/aws/glue/TestGlueCatalog.java | 47 +-
.../aws/glue/TestGlueToIcebergConverter.java | 12 +-
.../aws/glue/TestIcebergToGlueConverter.java | 137 ++-
.../org/apache/iceberg/aws/s3/TestS3FileIO.java | 56 +-
.../iceberg/aws/s3/TestS3FileIOProperties.java | 155 ++-
.../apache/iceberg/aws/s3/TestS3InputStream.java | 17 +-
.../apache/iceberg/aws/s3/TestS3OutputStream.java | 31 +-
.../apache/iceberg/aws/s3/TestS3RequestUtil.java | 53 +-
.../java/org/apache/iceberg/aws/s3/TestS3URI.java | 40 +-
.../iceberg/aws/s3/signer/S3SignerServlet.java | 5 +-
.../iceberg/aws/s3/signer/TestS3RestSigner.java | 6 +-
.../aws/s3/signer/TestS3SignRequestParser.java | 34 +-
.../aws/s3/signer/TestS3SignResponseParser.java | 19 +-
.../apache/iceberg/aws/util/TestRetryDetector.java | 37 +-
azure-bundle/NOTICE | 2 +-
azure-bundle/build.gradle | 2 +-
.../org/apache/iceberg/azure/AzureProperties.java | 9 +-
.../apache/iceberg/azure/AzurePropertiesTest.java | 30 +-
baseline.gradle | 49 +-
build.gradle | 15 +-
bundled-guava/NOTICE | 2 +-
.../java/org/apache/iceberg/common/DynClasses.java | 2 +-
.../org/apache/iceberg/common/DynConstructors.java | 34 +-
.../java/org/apache/iceberg/common/DynFields.java | 2 +-
.../java/org/apache/iceberg/common/DynMethods.java | 2 +-
.../apache/iceberg/common/TestDynConstructors.java | 75 ++
.../java/org/apache/iceberg/BaseEntriesTable.java | 187 +++-
.../java/org/apache/iceberg/BaseFilesTable.java | 2 +-
.../java/org/apache/iceberg/BaseMetadataTable.java | 5 +-
.../iceberg/BaseMetastoreTableOperations.java | 2 +-
.../org/apache/iceberg/BaseReplacePartitions.java | 2 +-
.../org/apache/iceberg/BaseRewriteManifests.java | 2 +-
.../java/org/apache/iceberg/BaseTransaction.java | 2 +-
.../main/java/org/apache/iceberg/CatalogUtil.java | 2 +-
.../java/org/apache/iceberg/ClientPoolImpl.java | 4 +-
.../main/java/org/apache/iceberg/DataFiles.java | 6 -
.../java/org/apache/iceberg/DataTaskParser.java | 81 ++
.../java/org/apache/iceberg/DeleteFileIndex.java | 16 -
.../main/java/org/apache/iceberg/FastAppend.java | 12 +-
.../org/apache/iceberg/FileScanTaskParser.java | 39 +-
.../java/org/apache/iceberg/ManifestReader.java | 2 +-
.../apache/iceberg/MergingSnapshotProducer.java | 21 +-
.../java/org/apache/iceberg/MetadataUpdate.java | 6 +-
.../main/java/org/apache/iceberg/MetricsModes.java | 2 +-
.../org/apache/iceberg/PositionDeletesTable.java | 37 +-
.../java/org/apache/iceberg/PropertiesUpdate.java | 2 +-
.../java/org/apache/iceberg/ReachableFileUtil.java | 19 -
.../java/org/apache/iceberg/ScanTaskParser.java | 105 ++
.../java/org/apache/iceberg/SnapshotProducer.java | 142 +--
.../java/org/apache/iceberg/StaticDataTask.java | 26 +
.../java/org/apache/iceberg/TableMetadata.java | 21 +-
.../iceberg/UpdateSnapshotReferencesOperation.java | 2 +-
.../actions/BaseRewriteDataFilesAction.java | 16 +-
.../actions/RewriteDataFilesActionResult.java | 4 +-
.../actions/RewriteDataFilesCommitManager.java | 9 +-
.../RewritePositionDeletesCommitManager.java | 9 +-
.../iceberg/avro/AvroCustomOrderSchemaVisitor.java | 2 +-
.../org/apache/iceberg/avro/AvroFileAppender.java | 8 +-
.../org/apache/iceberg/avro/AvroSchemaUtil.java | 7 +-
.../org/apache/iceberg/avro/AvroSchemaVisitor.java | 4 +-
.../iceberg/avro/AvroSchemaWithTypeVisitor.java | 2 +-
.../avro/AvroWithPartnerByStructureVisitor.java | 2 +-
.../iceberg/avro/AvroWithPartnerVisitor.java | 2 +-
.../org/apache/iceberg/avro/GenericAvroReader.java | 2 +-
.../java/org/apache/iceberg/avro/ValueWriters.java | 2 +-
.../encryption/NativeFileCryptoParameters.java | 6 +-
.../encryption/PlaintextEncryptionManager.java | 4 +-
.../org/apache/iceberg/hadoop/HadoopStreams.java | 5 -
.../iceberg/hadoop/HadoopTableOperations.java | 4 +-
.../org/apache/iceberg/jdbc/JdbcClientPool.java | 2 +-
.../org/apache/iceberg/rest/CatalogHandlers.java | 13 +-
.../apache/iceberg/rest/RESTSessionCatalog.java | 24 +-
.../org/apache/iceberg/rest/auth/OAuth2Util.java | 31 +-
.../iceberg/rest/responses/ErrorResponse.java | 8 +-
.../iceberg/rest/responses/LoadTableResponse.java | 2 +-
.../apache/iceberg/schema/UnionByNameVisitor.java | 2 +-
.../java/org/apache/iceberg/util/ArrayUtil.java | 14 +-
.../main/java/org/apache/iceberg/util/Pair.java | 6 +-
.../org/apache/iceberg/util/ParallelIterable.java | 6 +
.../main/java/org/apache/iceberg/util/Tasks.java | 4 +-
.../org/apache/iceberg/util/ZOrderByteUtils.java | 14 +-
.../apache/iceberg/view/BaseViewOperations.java | 2 +-
.../apache/iceberg/MetadataTableScanTestBase.java | 5 +-
.../apache/iceberg/TableMetadataParserTest.java | 5 +-
.../src/test/java/org/apache/iceberg/TestBase.java | 2 +-
.../iceberg/TestBaseIncrementalAppendScan.java | 26 +-
.../iceberg/TestBaseIncrementalChangelogScan.java | 4 +-
.../java/org/apache/iceberg/TestCatalogUtil.java | 60 +-
.../org/apache/iceberg/TestClientPoolImpl.java | 2 +-
.../org/apache/iceberg/TestContentFileParser.java | 59 +-
.../org/apache/iceberg/TestDataTaskParser.java | 274 ++++++
.../org/apache/iceberg/TestEnvironmentContext.java | 5 +-
.../java/org/apache/iceberg/TestFastAppend.java | 7 +
.../org/apache/iceberg/TestFileScanTaskParser.java | 89 +-
.../TestFixedSizeSplitScanTaskIterator.java | 7 +-
.../org/apache/iceberg/TestManifestReader.java | 4 +-
.../java/org/apache/iceberg/TestMergeAppend.java | 7 +
.../org/apache/iceberg/TestMetadataTableScans.java | 219 ++++-
...stMetadataTableScansWithPartitionEvolution.java | 9 +-
.../org/apache/iceberg/TestMetricsTruncation.java | 115 +--
.../TestOffsetsBasedSplitScanTaskIterator.java | 9 +-
.../java/org/apache/iceberg/TestRewriteFiles.java | 6 +-
.../org/apache/iceberg/TestScanTaskParser.java | 42 +-
.../apache/iceberg/TestSchemaUnionByFieldName.java | 58 +-
.../org/apache/iceberg/TestSnapshotSummary.java | 275 +++++-
.../java/org/apache/iceberg/TestTableMetadata.java | 47 +-
.../iceberg/TestUpdateRequirementParser.java | 49 +-
.../apache/iceberg/TestV1ToV2RowDeltaDelete.java | 6 +-
.../apache/iceberg/avro/TestAvroNameMapping.java | 43 +-
.../iceberg/avro/TestAvroReadProjection.java | 7 +-
.../iceberg/avro/TestBuildAvroProjection.java | 47 +-
.../avro/TestNameMappingWithAvroSchema.java | 68 +-
.../apache/iceberg/avro/TestReadProjection.java | 230 ++---
.../org/apache/iceberg/catalog/CatalogTests.java | 717 +++++++-------
.../iceberg/catalog/TestTableIdentifierParser.java | 28 +-
.../iceberg/data/avro/TestDecoderResolver.java | 2 +
.../iceberg/encryption/KeyStoreKmsClient.java | 5 +-
.../org/apache/iceberg/encryption/TestCiphers.java | 6 +-
.../iceberg/expressions/TestExpressionParser.java | 100 +-
.../apache/iceberg/hadoop/HadoopFileIOTest.java | 35 +-
.../apache/iceberg/hadoop/TestCachingCatalog.java | 119 ++-
.../iceberg/hadoop/TestCatalogUtilDropTable.java | 35 +-
.../apache/iceberg/hadoop/TestHadoopCatalog.java | 196 ++--
.../apache/iceberg/hadoop/TestHadoopCommits.java | 159 ++--
.../apache/iceberg/hadoop/TestHadoopStreams.java | 5 +-
.../apache/iceberg/hadoop/TestHadoopTables.java | 51 +-
.../org/apache/iceberg/hadoop/TestStaticTable.java | 29 +-
.../iceberg/hadoop/TestTableSerialization.java | 15 +-
.../iceberg/inmemory/TestInMemoryFileIO.java | 28 +-
.../iceberg/inmemory/TestInMemoryInputFile.java | 8 +-
.../iceberg/inmemory/TestInMemoryOutputFile.java | 9 +-
.../org/apache/iceberg/io/MockInputStream.java | 2 +-
.../iceberg/io/TestByteBufferInputStreams.java | 214 ++---
.../java/org/apache/iceberg/io/TestIOUtil.java | 52 +-
.../org/apache/iceberg/jdbc/TestJdbcCatalog.java | 57 +-
.../iceberg/metrics/TestCommitReportParser.java | 35 +-
.../iceberg/metrics/TestCounterResultParser.java | 23 +-
.../metrics/TestScanMetricsResultParser.java | 44 +-
.../org/apache/iceberg/metrics/TestScanReport.java | 70 +-
.../iceberg/metrics/TestScanReportParser.java | 47 +-
.../iceberg/metrics/TestTimerResultParser.java | 62 +-
.../apache/iceberg/rest/RESTCatalogAdapter.java | 1 -
.../iceberg/rest/RequestResponseTestBase.java | 11 +-
.../org/apache/iceberg/rest/TestHTTPClient.java | 18 +-
.../org/apache/iceberg/rest/TestRESTCatalog.java | 138 ++-
.../java/org/apache/iceberg/rest/TestRESTUtil.java | 22 +-
.../org/apache/iceberg/rest/TestResourcePaths.java | 97 +-
.../rest/requests/TestCreateNamespaceRequest.java | 29 +-
.../rest/requests/TestCreateTableRequest.java | 44 +-
.../requests/TestRegisterTableRequestParser.java | 13 +-
.../rest/requests/TestRenameTableRequest.java | 24 +-
.../requests/TestReportMetricsRequestParser.java | 30 +-
.../TestUpdateNamespacePropertiesRequest.java | 36 +-
.../responses/TestCatalogErrorResponseParser.java | 15 +-
.../iceberg/rest/responses/TestConfigResponse.java | 32 +-
.../responses/TestCreateNamespaceResponse.java | 28 +-
.../rest/responses/TestGetNamespaceResponse.java | 27 +-
.../rest/responses/TestListNamespacesResponse.java | 29 +-
.../rest/responses/TestListTablesResponse.java | 34 +-
.../rest/responses/TestLoadTableResponse.java | 77 +-
.../responses/TestOAuthErrorResponseParser.java | 12 +-
.../rest/responses/TestOAuthTokenResponse.java | 26 +-
.../TestUpdateNamespacePropertiesResponse.java | 36 +-
.../iceberg/util/TestInMemoryLockManager.java | 4 +-
.../java/org/apache/iceberg/util/TestJsonUtil.java | 247 ++---
.../org/apache/iceberg/util/TestLocationUtil.java | 4 +-
.../org/apache/iceberg/util/TestLockManagers.java | 7 +-
.../apache/iceberg/util/TestParallelIterable.java | 61 ++
.../org/apache/iceberg/util/TestSnapshotUtil.java | 3 +-
.../org/apache/iceberg/util/TestTruncateUtil.java | 8 +-
.../view/TestSQLViewRepresentationParser.java | 20 +-
.../iceberg/view/TestViewHistoryEntryParser.java | 19 +-
.../iceberg/view/TestViewRepresentationParser.java | 13 +-
.../apache/iceberg/view/TestViewVersionParser.java | 16 +-
.../apache/iceberg/data/TableMigrationUtil.java | 57 +-
.../org/apache/iceberg/data/DataTestHelpers.java | 23 +-
.../org/apache/iceberg/data/TestGenericRecord.java | 4 +-
.../org/apache/iceberg/data/TestLocalScan.java | 8 +-
.../iceberg/data/TestMetricsRowGroupFilter.java | 9 +-
.../apache/iceberg/data/TestReadProjection.java | 6 +-
.../data/avro/TestSingleMessageEncoding.java | 14 +-
.../org/apache/iceberg/io/TestBaseTaskWriter.java | 7 +-
.../java/org/apache/iceberg/dell/ecs/EcsURI.java | 5 +-
.../apache/iceberg/dell/ecs/TestEcsCatalog.java | 22 +-
.../apache/iceberg/dell/ecs/TestEcsOutputFile.java | 4 +-
.../iceberg/dell/ecs/TestEcsTableOperations.java | 4 +-
.../org/apache/iceberg/dell/ecs/TestEcsURI.java | 4 +-
.../iceberg/dell/mock/ecs/EcsS3MockRule.java | 2 +-
.../apache/iceberg/dell/mock/ecs/ObjectData.java | 2 +-
.../iceberg/dell/mock/ecs/TestExceptionCode.java | 16 +-
.../iceberg/delta/TestSnapshotDeltaLakeTable.java | 42 +-
.../TestBaseSnapshotDeltaLakeTableAction.java | 11 +-
.../iceberg/delta/TestDeltaLakeTypeToType.java | 64 +-
docs/docs/flink-connector.md | 4 +-
docs/docs/flink-ddl.md | 4 +-
docs/docs/flink-queries.md | 4 +-
docs/docs/flink.md | 22 +-
docs/docs/spark-procedures.md | 4 +-
docs/mkdocs.yml | 1 +
flink/v1.17/build.gradle | 2 +-
flink/v1.17/flink-runtime/NOTICE | 2 +-
.../sink/shuffle/MapRangePartitionerBenchmark.java | 4 +-
.../org/apache/iceberg/flink/actions/Actions.java | 4 +-
.../flink/actions/RewriteDataFilesAction.java | 2 +-
.../iceberg/flink/data/FlinkParquetReaders.java | 6 +-
.../sink/AvroGenericRecordToRowDataMapper.java | 2 +-
.../flink/sink/shuffle/MapDataStatistics.java | 2 +-
.../flink/sink/shuffle/MapRangePartitioner.java | 8 +-
.../RowDataToAvroGenericRecordConverter.java | 2 +-
.../enumerator/ContinuousSplitPlannerImpl.java | 2 +-
.../source/enumerator/EnumerationHistory.java | 6 +-
.../source/enumerator/IcebergEnumeratorState.java | 2 +-
.../IcebergEnumeratorStateSerializer.java | 6 +-
.../flink/source/split/IcebergSourceSplit.java | 6 +-
.../flink/source/split/SerializerHelper.java | 6 +-
.../apache/iceberg/flink/util/FlinkPackage.java | 2 +-
.../iceberg/flink/HadoopCatalogExtension.java | 5 +-
.../iceberg/flink/HadoopTableExtension.java} | 23 +-
.../java/org/apache/iceberg/flink/TestBase.java | 10 +-
.../iceberg/flink/TestCatalogTableLoader.java | 15 +-
.../iceberg/flink/TestFlinkCatalogDatabase.java | 7 +-
.../iceberg/flink/TestFlinkCatalogTable.java | 31 +-
.../flink/TestFlinkCatalogTablePartitions.java | 6 +-
.../java/org/apache/iceberg/flink/TestHelpers.java | 3 +-
.../org/apache/iceberg/flink/TestTableLoader.java | 2 +-
.../flink/actions/TestRewriteDataFilesAction.java | 4 +-
.../iceberg/flink/data/TestRowDataProjection.java | 60 +-
.../flink/sink/TestBucketPartitionKeySelector.java | 8 +-
.../iceberg/flink/sink/TestBucketPartitioner.java | 15 +-
.../TestBucketPartitionerFlinkIcebergSink.java | 14 +-
.../iceberg/flink/sink/TestFlinkIcebergSink.java | 157 ++-
.../flink/sink/TestFlinkIcebergSinkBranch.java | 66 +-
.../iceberg/flink/sink/TestFlinkIcebergSinkV2.java | 117 +--
.../flink/sink/TestFlinkIcebergSinkV2Base.java | 49 +-
.../flink/sink/TestFlinkIcebergSinkV2Branch.java | 69 +-
.../flink/sink/TestIcebergFilesCommitter.java | 4 +-
.../flink/sink/TestIcebergStreamWriter.java | 8 +-
.../flink/sink/shuffle/TestMapDataStatistics.java | 5 +-
.../sink/shuffle/TestMapRangePartitioner.java | 25 +-
.../iceberg/flink/source/TestFlinkSourceSql.java | 4 +-
.../iceberg/flink/source/TestIcebergSourceSql.java | 4 +-
.../iceberg/flink/source/TestScanContext.java | 5 +-
.../apache/iceberg/flink/source/TestSqlBase.java | 46 +-
.../iceberg/flink/source/TestStreamScanSql.java | 44 +-
.../TestFileSequenceNumberBasedSplitAssigner.java | 5 +-
.../assigner/TestWatermarkBasedSplitAssigner.java | 2 +-
.../TestContinuousIcebergEnumerator.java | 10 +-
.../enumerator/TestContinuousSplitPlannerImpl.java | 11 +-
...estContinuousSplitPlannerImplStartStrategy.java | 7 +-
.../iceberg/flink/source/reader/ReaderUtil.java | 24 +-
.../reader/TestColumnStatsWatermarkExtractor.java | 96 +-
.../source/reader/TestIcebergSourceReader.java | 2 +-
.../source/reader/TestRowDataReaderFunction.java | 2 +-
flink/v1.18/build.gradle | 2 +-
flink/v1.18/flink-runtime/NOTICE | 2 +-
.../sink/shuffle/MapRangePartitionerBenchmark.java | 4 +-
.../org/apache/iceberg/flink/actions/Actions.java | 4 +-
.../flink/actions/RewriteDataFilesAction.java | 2 +-
.../iceberg/flink/data/FlinkParquetReaders.java | 6 +-
.../sink/AvroGenericRecordToRowDataMapper.java | 2 +-
.../flink/sink/shuffle/MapDataStatistics.java | 2 +-
.../flink/sink/shuffle/MapRangePartitioner.java | 8 +-
.../RowDataToAvroGenericRecordConverter.java | 2 +-
.../enumerator/ContinuousSplitPlannerImpl.java | 2 +-
.../source/enumerator/EnumerationHistory.java | 6 +-
.../source/enumerator/IcebergEnumeratorState.java | 2 +-
.../IcebergEnumeratorStateSerializer.java | 6 +-
.../flink/source/split/IcebergSourceSplit.java | 6 +-
.../flink/source/split/SerializerHelper.java | 6 +-
.../apache/iceberg/flink/util/FlinkPackage.java | 2 +-
.../iceberg/flink/HadoopCatalogExtension.java | 5 +-
...ableResource.java => HadoopTableExtension.java} | 23 +-
.../java/org/apache/iceberg/flink/TestBase.java | 10 +-
.../iceberg/flink/TestCatalogTableLoader.java | 15 +-
.../iceberg/flink/TestFlinkCatalogDatabase.java | 7 +-
.../iceberg/flink/TestFlinkCatalogTable.java | 31 +-
.../flink/TestFlinkCatalogTablePartitions.java | 6 +-
.../java/org/apache/iceberg/flink/TestHelpers.java | 3 +-
.../org/apache/iceberg/flink/TestTableLoader.java | 2 +-
.../flink/actions/TestRewriteDataFilesAction.java | 4 +-
.../iceberg/flink/data/TestRowDataProjection.java | 60 +-
.../flink/sink/TestBucketPartitionKeySelector.java | 8 +-
.../iceberg/flink/sink/TestBucketPartitioner.java | 15 +-
.../TestBucketPartitionerFlinkIcebergSink.java | 14 +-
.../iceberg/flink/sink/TestFlinkIcebergSink.java | 157 ++-
.../flink/sink/TestFlinkIcebergSinkBranch.java | 66 +-
.../iceberg/flink/sink/TestFlinkIcebergSinkV2.java | 117 +--
.../flink/sink/TestFlinkIcebergSinkV2Base.java | 49 +-
.../flink/sink/TestFlinkIcebergSinkV2Branch.java | 69 +-
.../flink/sink/TestIcebergFilesCommitter.java | 4 +-
.../flink/sink/TestIcebergStreamWriter.java | 8 +-
.../flink/sink/shuffle/TestMapDataStatistics.java | 5 +-
.../sink/shuffle/TestMapRangePartitioner.java | 25 +-
.../iceberg/flink/source/TestFlinkSourceSql.java | 4 +-
.../iceberg/flink/source/TestIcebergSourceSql.java | 4 +-
.../iceberg/flink/source/TestScanContext.java | 5 +-
.../apache/iceberg/flink/source/TestSqlBase.java | 46 +-
.../iceberg/flink/source/TestStreamScanSql.java | 44 +-
.../TestFileSequenceNumberBasedSplitAssigner.java | 5 +-
.../assigner/TestWatermarkBasedSplitAssigner.java | 2 +-
.../TestContinuousIcebergEnumerator.java | 10 +-
.../enumerator/TestContinuousSplitPlannerImpl.java | 11 +-
...estContinuousSplitPlannerImplStartStrategy.java | 7 +-
.../iceberg/flink/source/reader/ReaderUtil.java | 24 +-
.../reader/TestColumnStatsWatermarkExtractor.java | 96 +-
.../source/reader/TestIcebergSourceReader.java | 2 +-
.../source/reader/TestRowDataReaderFunction.java | 2 +-
flink/v1.19/build.gradle | 2 +-
flink/v1.19/flink-runtime/NOTICE | 2 +-
.../org/apache/iceberg/flink/actions/Actions.java | 4 +-
.../flink/actions/RewriteDataFilesAction.java | 2 +-
.../iceberg/flink/data/FlinkParquetReaders.java | 6 +-
.../sink/AvroGenericRecordToRowDataMapper.java | 2 +-
.../flink/sink/shuffle/MapDataStatistics.java | 2 +-
.../flink/sink/shuffle/MapRangePartitioner.java | 8 +-
.../RowDataToAvroGenericRecordConverter.java | 2 +-
.../enumerator/ContinuousSplitPlannerImpl.java | 2 +-
.../source/enumerator/EnumerationHistory.java | 6 +-
.../source/enumerator/IcebergEnumeratorState.java | 2 +-
.../IcebergEnumeratorStateSerializer.java | 6 +-
.../flink/source/split/IcebergSourceSplit.java | 6 +-
.../flink/source/split/SerializerHelper.java | 6 +-
.../apache/iceberg/flink/util/FlinkPackage.java | 2 +-
.../iceberg/flink/HadoopCatalogExtension.java | 5 +-
.../iceberg/flink/HadoopTableExtension.java} | 23 +-
.../java/org/apache/iceberg/flink/TestBase.java | 9 +-
.../iceberg/flink/TestCatalogTableLoader.java | 15 +-
.../iceberg/flink/TestFlinkCatalogDatabase.java | 7 +-
.../iceberg/flink/TestFlinkCatalogTable.java | 31 +-
.../flink/TestFlinkCatalogTablePartitions.java | 6 +-
.../java/org/apache/iceberg/flink/TestHelpers.java | 3 +-
.../org/apache/iceberg/flink/TestTableLoader.java | 2 +-
.../flink/actions/TestRewriteDataFilesAction.java | 4 +-
.../iceberg/flink/data/TestRowDataProjection.java | 60 +-
.../maintenance/operator/FlinkSqlExtension.java | 5 +-
.../flink/sink/TestBucketPartitionKeySelector.java | 8 +-
.../iceberg/flink/sink/TestBucketPartitioner.java | 15 +-
.../TestBucketPartitionerFlinkIcebergSink.java | 14 +-
.../iceberg/flink/sink/TestFlinkIcebergSink.java | 157 ++-
.../flink/sink/TestFlinkIcebergSinkBranch.java | 66 +-
.../iceberg/flink/sink/TestFlinkIcebergSinkV2.java | 117 +--
.../flink/sink/TestFlinkIcebergSinkV2Base.java | 49 +-
.../flink/sink/TestFlinkIcebergSinkV2Branch.java | 69 +-
.../flink/sink/TestIcebergFilesCommitter.java | 4 +-
.../flink/sink/TestIcebergStreamWriter.java | 8 +-
.../flink/sink/shuffle/TestMapDataStatistics.java | 4 +-
.../sink/shuffle/TestMapRangePartitioner.java | 25 +-
.../sink/shuffle/TestSketchDataStatistics.java | 4 +-
.../apache/iceberg/flink/source/SplitHelpers.java | 21 +-
.../iceberg/flink/source/TestFlinkSourceSql.java | 4 +-
.../iceberg/flink/source/TestIcebergSourceSql.java | 4 +-
.../iceberg/flink/source/TestScanContext.java | 5 +-
.../apache/iceberg/flink/source/TestSqlBase.java | 46 +-
.../iceberg/flink/source/TestStreamScanSql.java | 46 +-
.../source/assigner/SplitAssignerTestBase.java | 28 +-
.../source/assigner/TestDefaultSplitAssigner.java | 4 +-
.../TestFileSequenceNumberBasedSplitAssigner.java | 13 +-
.../assigner/TestWatermarkBasedSplitAssigner.java | 12 +-
.../TestContinuousIcebergEnumerator.java | 107 +--
.../enumerator/TestContinuousSplitPlannerImpl.java | 362 +++----
...estContinuousSplitPlannerImplStartStrategy.java | 82 +-
.../source/enumerator/TestEnumerationHistory.java | 23 +-
.../TestIcebergEnumeratorStateSerializer.java | 67 +-
.../source/reader/ReaderFunctionTestBase.java | 66 +-
.../iceberg/flink/source/reader/ReaderUtil.java | 11 +-
.../flink/source/reader/TestArrayBatchRecords.java | 19 +-
.../TestArrayPoolDataIteratorBatcherRowData.java | 242 ++---
.../reader/TestColumnStatsWatermarkExtractor.java | 96 +-
.../source/reader/TestIcebergSourceReader.java | 35 +-
.../source/reader/TestRowDataReaderFunction.java | 15 +-
.../split/TestIcebergSourceSplitSerializer.java | 47 +-
format/spec.md | 7 +-
gcp-bundle/NOTICE | 2 +-
gcp-bundle/build.gradle | 2 +-
.../org/apache/iceberg/gcp/GCPPropertiesTest.java | 4 +-
.../apache/iceberg/gcp/gcs/GCSInputStreamTest.java | 3 +-
.../apache/iceberg/gcp/gcs/GCSLocationTest.java | 24 +-
gradle.properties | 7 +
gradle/libs.versions.toml | 20 +-
gradle/wrapper/gradle-wrapper.properties | 5 +-
gradlew | 24 +-
.../org/apache/iceberg/hive/CachedClientPool.java | 2 +-
.../org/apache/iceberg/hive/HiveClientPool.java | 3 +-
.../apache/iceberg/hive/HiveSchemaConverter.java | 2 +-
.../java/org/apache/iceberg/hive/ScriptRunner.java | 14 +-
.../org/apache/iceberg/hive/TestHiveCatalog.java | 2 +-
.../apache/iceberg/hive/TestHiveCommitLocks.java | 4 +-
hive-runtime/NOTICE | 2 +-
hive-runtime/build.gradle | 2 +-
hive3-orc-bundle/build.gradle | 2 +-
.../hive/ql/exec/vector/VectorizedSupport.java | 3 +-
.../main/java/org/apache/iceberg/mr/Catalogs.java | 2 +-
.../org/apache/iceberg/mr/hive/Deserializer.java | 6 +-
.../iceberg/mr/hive/HiveIcebergMetaHook.java | 3 +-
.../apache/iceberg/mr/hive/HiveIcebergSerDe.java | 5 +-
.../IcebergRecordObjectInspector.java | 3 +-
.../java/org/apache/iceberg/mr/TestHelper.java | 4 +-
.../iceberg/mr/hive/HiveIcebergTestUtils.java | 14 +-
.../mr/hive/TestHiveIcebergFilterFactory.java | 4 +-
.../mr/hive/TestHiveIcebergOutputCommitter.java | 5 +-
.../TestHiveIcebergStorageHandlerLocalScan.java | 2 +-
.../hive/TestHiveIcebergStorageHandlerNoScan.java | 3 +-
.../TestHiveIcebergStorageHandlerWithEngine.java | 30 +-
.../org/apache/iceberg/mr/hive/TestTables.java | 5 +-
.../apache/iceberg/nessie/NessieIcebergClient.java | 15 +-
.../org/apache/iceberg/nessie/BaseTestIceberg.java | 2 +-
.../iceberg/nessie/TestBranchVisibility.java | 86 +-
.../apache/iceberg/nessie/TestMultipleClients.java | 59 +-
.../org/apache/iceberg/nessie/TestNamespace.java | 79 +-
.../apache/iceberg/nessie/TestNessieCatalog.java | 5 +-
.../iceberg/nessie/TestNessieIcebergClient.java | 131 ++-
.../org/apache/iceberg/nessie/TestNessieTable.java | 169 ++--
.../org/apache/iceberg/nessie/TestNessieUtil.java | 29 +-
.../org/apache/iceberg/nessie/TestNessieView.java | 65 +-
open-api/requirements.txt | 2 +-
.../apache/iceberg/data/orc/GenericOrcReaders.java | 2 +-
.../iceberg/orc/ExpressionToSearchArgument.java | 6 +-
orc/src/main/java/org/apache/iceberg/orc/ORC.java | 2 +-
.../java/org/apache/iceberg/orc/ORCSchemaUtil.java | 6 +-
.../org/apache/iceberg/orc/TestBloomFilter.java | 20 +-
.../apache/iceberg/orc/TestBuildOrcProjection.java | 69 +-
.../orc/TestEstimateOrcAvgWidthVisitor.java | 64 +-
.../orc/TestExpressionToSearchArgument.java | 26 +-
.../org/apache/iceberg/orc/TestIdToOrcName.java | 54 +-
.../apache/iceberg/orc/TestORCFileIOProxies.java | 22 +-
.../org/apache/iceberg/orc/TestORCSchemaUtil.java | 44 +-
.../org/apache/iceberg/orc/TestOrcDataWriter.java | 38 +-
.../apache/iceberg/orc/TestOrcDeleteWriters.java | 41 +-
.../apache/iceberg/orc/TestTableProperties.java | 37 +-
.../iceberg/data/parquet/BaseParquetReaders.java | 8 +-
.../java/org/apache/iceberg/parquet/Parquet.java | 4 +-
.../org/apache/iceberg/parquet/ParquetAvro.java | 6 +-
.../parquet/ParquetBloomRowGroupFilter.java | 2 +-
.../apache/iceberg/parquet/ParquetTypeVisitor.java | 2 +-
.../iceberg/parquet/ParquetValueReaders.java | 44 +-
.../iceberg/parquet/ParquetValueWriters.java | 10 +-
.../iceberg/parquet/ParquetWriteAdapter.java | 2 +-
.../iceberg/parquet/VectorizedParquetReader.java | 2 +-
.../iceberg/parquet/ParquetWritingTestUtils.java | 2 +-
.../iceberg/parquet/TestBloomRowGroupFilter.java | 2 +-
.../parquet/TestDictionaryRowGroupFilter.java | 2 +-
.../org/apache/iceberg/pig/IcebergStorage.java | 4 +-
settings.gradle | 15 +-
site/docs/contribute.md | 2 +-
site/mkdocs.yml | 2 +
site/requirements.txt | 2 +-
.../iceberg/snowflake/JdbcSnowflakeClient.java | 10 +-
.../apache/iceberg/snowflake/SnowflakeCatalog.java | 2 +-
.../iceberg/snowflake/SnowflakeTableMetadata.java | 2 +-
.../iceberg/snowflake/JdbcSnowflakeClientTest.java | 89 +-
.../iceberg/snowflake/NamespaceHelpersTest.java | 31 +-
.../iceberg/snowflake/SnowflakeCatalogTest.java | 62 +-
spark/v3.3/build.gradle | 2 +-
.../spark/extensions/TestAddFilesProcedure.java | 114 +--
.../extensions/TestAlterTablePartitionFields.java | 27 +-
.../spark/extensions/TestAlterTableSchema.java | 52 +-
.../spark/extensions/TestAncestorsOfProcedure.java | 34 +-
.../iceberg/spark/extensions/TestBranchDDL.java | 144 ++-
.../spark/extensions/TestCallStatementParser.java | 12 +-
.../spark/extensions/TestChangelogTable.java | 10 +-
.../TestCherrypickSnapshotProcedure.java | 66 +-
.../spark/extensions/TestConflictValidation.java | 292 +++---
.../spark/extensions/TestCopyOnWriteDelete.java | 4 +-
.../spark/extensions/TestCopyOnWriteMerge.java | 4 +-
.../spark/extensions/TestCopyOnWriteUpdate.java | 4 +-
.../iceberg/spark/extensions/TestDelete.java | 39 +-
.../extensions/TestExpireSnapshotsProcedure.java | 160 ++--
.../apache/iceberg/spark/extensions/TestMerge.java | 1003 ++++++++++----------
.../spark/extensions/TestMergeOnReadDelete.java | 16 +-
.../TestMetaColumnProjectionWithStageScan.java | 5 +-
.../extensions/TestMigrateTableProcedure.java | 50 +-
.../extensions/TestPublishChangesProcedure.java | 57 +-
.../extensions/TestRemoveOrphanFilesProcedure.java | 197 ++--
.../spark/extensions/TestReplaceBranch.java | 32 +-
.../TestRequiredDistributionAndOrdering.java | 31 +-
.../extensions/TestRewriteDataFilesProcedure.java | 227 +++--
.../extensions/TestRewriteManifestsProcedure.java | 68 +-
.../TestRewritePositionDeleteFilesProcedure.java | 6 +-
.../TestRollbackToSnapshotProcedure.java | 95 +-
.../TestRollbackToTimestampProcedure.java | 95 +-
.../TestSetCurrentSnapshotProcedure.java | 106 +--
.../TestSetWriteDistributionAndOrdering.java | 7 +-
.../extensions/TestSnapshotTableProcedure.java | 64 +-
.../iceberg/spark/extensions/TestTagDDL.java | 131 ++-
.../iceberg/spark/extensions/TestUpdate.java | 152 ++-
.../iceberg/spark/extensions/TestWriteAborts.java | 37 +-
spark/v3.3/spark-runtime/NOTICE | 2 +-
.../iceberg/spark/action/RandomGeneratingUDF.java | 5 +-
.../iceberg/spark/PruneColumnsWithReordering.java | 2 +-
.../java/org/apache/iceberg/spark/Spark3Util.java | 2 +-
.../org/apache/iceberg/spark/SparkTableUtil.java | 14 +-
.../actions/DeleteOrphanFilesSparkAction.java | 2 +-
.../spark/actions/RewriteDataFilesSparkAction.java | 2 +-
.../spark/actions/RewriteManifestsSparkAction.java | 28 +-
.../RewritePositionDeleteFilesSparkAction.java | 15 +-
.../iceberg/spark/data/SparkParquetWriters.java | 2 +-
.../spark/source/SparkPositionDeltaWrite.java | 10 +-
.../iceberg/spark/source/SparkScanBuilder.java | 2 +-
.../spark/source/SparkStagedScanBuilder.java | 2 +-
.../apache/iceberg/spark/source/SparkWrite.java | 10 +-
.../apache/iceberg/TestDataFileSerialization.java | 6 +-
.../iceberg/TestManifestFileSerialization.java | 6 +-
.../apache/iceberg/TestScanTaskSerialization.java | 14 +-
.../java/org/apache/iceberg/ValidationHelpers.java | 5 +-
.../apache/iceberg/spark/TestFunctionCatalog.java | 52 +-
.../apache/iceberg/spark/TestSparkTableUtil.java | 7 +-
.../iceberg/spark/actions/TestCreateActions.java | 2 +-
.../actions/TestDeleteReachableFilesAction.java | 21 +-
.../spark/actions/TestExpireSnapshotsAction.java | 22 +-
.../spark/actions/TestRemoveOrphanFilesAction.java | 71 +-
.../spark/actions/TestRewriteDataFilesAction.java | 176 ++--
.../spark/actions/TestRewriteManifestsAction.java | 13 +-
.../TestRewritePositionDeleteFilesAction.java | 66 ++
.../spark/actions/TestSparkFileRewriter.java | 47 +-
.../apache/iceberg/spark/data/GenericsHelpers.java | 100 +-
.../org/apache/iceberg/spark/data/RandomData.java | 4 +-
.../org/apache/iceberg/spark/data/TestHelpers.java | 110 +--
.../iceberg/spark/data/TestSparkParquetReader.java | 4 +-
.../vectorized/TestParquetVectorizedReads.java | 38 +-
.../apache/iceberg/spark/source/LogMessage.java | 2 +-
.../iceberg/spark/source/TestBaseReader.java | 2 +-
.../spark/source/TestDataFrameWriterV2.java | 51 +-
.../iceberg/spark/source/TestDataFrameWrites.java | 38 +-
.../spark/source/TestDataSourceOptions.java | 90 +-
.../iceberg/spark/source/TestFilteredScan.java | 4 +-
.../spark/source/TestForwardCompatibility.java | 31 +-
.../spark/source/TestIcebergSourceTablesBase.java | 30 +-
.../iceberg/spark/source/TestIcebergSpark.java | 9 +-
.../spark/source/TestIdentityPartitionData.java | 2 +-
.../TestMetadataTablesWithPartitionEvolution.java | 11 +-
.../iceberg/spark/source/TestPartitionPruning.java | 2 +-
.../iceberg/spark/source/TestPathIdentifier.java | 7 +-
.../iceberg/spark/source/TestReadProjection.java | 5 +-
.../TestRequiredDistributionAndOrdering.java | 53 +-
.../spark/source/TestSnapshotSelection.java | 23 +-
.../source/TestSparkCatalogCacheExpiration.java | 24 +-
.../iceberg/spark/source/TestSparkDataWrite.java | 35 +-
.../spark/source/TestSparkMetadataColumns.java | 20 +-
.../spark/source/TestStructuredStreaming.java | 10 +-
.../spark/source/TestStructuredStreamingRead3.java | 59 +-
.../spark/source/TestTimestampWithoutZone.java | 42 +-
.../spark/source/TestWriteMetricsConfig.java | 13 +-
.../iceberg/spark/sql/TestAggregatePushDown.java | 17 +-
.../apache/iceberg/spark/sql/TestAlterTable.java | 40 +-
.../apache/iceberg/spark/sql/TestCreateTable.java | 47 +-
.../apache/iceberg/spark/sql/TestDeleteFrom.java | 12 +-
.../apache/iceberg/spark/sql/TestDropTable.java | 13 +-
.../iceberg/spark/sql/TestFilterPushDown.java | 11 +-
.../apache/iceberg/spark/sql/TestNamespaceSQL.java | 12 +-
.../sql/TestPartitionedWritesToWapBranch.java | 8 +-
.../org/apache/iceberg/spark/sql/TestSelect.java | 115 ++-
.../iceberg/spark/sql/TestSparkBucketFunction.java | 202 ++--
.../iceberg/spark/sql/TestSparkDaysFunction.java | 43 +-
.../iceberg/spark/sql/TestSparkHoursFunction.java | 43 +-
.../iceberg/spark/sql/TestSparkMonthsFunction.java | 49 +-
.../spark/sql/TestSparkTruncateFunction.java | 195 ++--
.../iceberg/spark/sql/TestSparkYearsFunction.java | 49 +-
.../spark/sql/TestTimestampWithoutZone.java | 19 +-
.../spark/sql/TestUnpartitionedWritesToBranch.java | 5 +-
.../spark/sql/UnpartitionedWritesTestBase.java | 30 +-
spark/v3.4/build.gradle | 2 +-
.../SparkRowLevelOperationsTestBase.java | 2 +-
.../spark/extensions/TestAddFilesProcedure.java | 14 +-
.../extensions/TestAlterTablePartitionFields.java | 27 +-
.../spark/extensions/TestAlterTableSchema.java | 17 +-
.../spark/extensions/TestAncestorsOfProcedure.java | 10 +-
.../iceberg/spark/extensions/TestBranchDDL.java | 65 +-
.../spark/extensions/TestCallStatementParser.java | 5 +-
.../spark/extensions/TestChangelogTable.java | 5 +-
.../TestCherrypickSnapshotProcedure.java | 17 +-
.../spark/extensions/TestConflictValidation.java | 21 +-
.../spark/extensions/TestCopyOnWriteDelete.java | 4 +-
.../spark/extensions/TestCopyOnWriteMerge.java | 4 +-
.../spark/extensions/TestCopyOnWriteUpdate.java | 4 +-
.../iceberg/spark/extensions/TestDelete.java | 33 +-
.../extensions/TestExpireSnapshotsProcedure.java | 45 +-
.../apache/iceberg/spark/extensions/TestMerge.java | 91 +-
.../spark/extensions/TestMergeOnReadDelete.java | 5 +-
.../TestMetaColumnProjectionWithStageScan.java | 5 +-
.../extensions/TestMigrateTableProcedure.java | 17 +-
.../extensions/TestPublishChangesProcedure.java | 13 +-
.../extensions/TestRemoveOrphanFilesProcedure.java | 56 +-
.../spark/extensions/TestReplaceBranch.java | 7 +-
.../TestRequiredDistributionAndOrdering.java | 5 +-
.../extensions/TestRewriteDataFilesProcedure.java | 35 +-
.../extensions/TestRewriteManifestsProcedure.java | 16 +-
.../TestRewritePositionDeleteFilesProcedure.java | 6 +-
.../TestRollbackToSnapshotProcedure.java | 24 +-
.../TestRollbackToTimestampProcedure.java | 20 +-
.../TestSetCurrentSnapshotProcedure.java | 29 +-
.../TestSetWriteDistributionAndOrdering.java | 4 +-
.../extensions/TestSnapshotTableProcedure.java | 13 +-
...toragePartitionedJoinsInRowLevelOperations.java | 10 +-
.../extensions/TestSystemFunctionPushDownDQL.java | 28 +-
.../iceberg/spark/extensions/TestTagDDL.java | 28 +-
.../iceberg/spark/extensions/TestUpdate.java | 42 +-
.../iceberg/spark/extensions/TestWriteAborts.java | 5 +-
spark/v3.4/spark-runtime/NOTICE | 2 +-
.../java/org/apache/iceberg/spark/SmokeTest.java | 5 +-
.../iceberg/spark/action/RandomGeneratingUDF.java | 5 +-
.../iceberg/spark/PruneColumnsWithReordering.java | 2 +-
.../java/org/apache/iceberg/spark/Spark3Util.java | 2 +-
.../org/apache/iceberg/spark/SparkTableUtil.java | 2 +-
.../actions/DeleteOrphanFilesSparkAction.java | 2 +-
.../spark/actions/RewriteDataFilesSparkAction.java | 2 +-
.../spark/actions/RewriteManifestsSparkAction.java | 28 +-
.../RewritePositionDeleteFilesSparkAction.java | 16 +-
.../iceberg/spark/data/SparkParquetWriters.java | 2 +-
.../spark/source/SparkFileWriterFactory.java | 2 +-
.../spark/source/SparkPositionDeltaWrite.java | 10 +-
.../iceberg/spark/source/SparkScanBuilder.java | 91 +-
.../spark/source/SparkStagedScanBuilder.java | 2 +-
.../apache/iceberg/spark/source/SparkWrite.java | 10 +-
.../analysis/NoSuchProcedureException.java | 2 +-
.../apache/iceberg/TestDataFileSerialization.java | 6 +-
.../iceberg/TestManifestFileSerialization.java | 6 +-
.../apache/iceberg/TestScanTaskSerialization.java | 14 +-
.../java/org/apache/iceberg/ValidationHelpers.java | 5 +-
.../apache/iceberg/spark/TestFunctionCatalog.java | 18 +-
.../org/apache/iceberg/spark/TestSpark3Util.java | 19 +-
.../apache/iceberg/spark/TestSparkTableUtil.java | 7 +-
.../apache/iceberg/spark/TestSparkV2Filters.java | 32 +-
.../iceberg/spark/actions/TestCreateActions.java | 2 +-
.../actions/TestDeleteReachableFilesAction.java | 7 +-
.../spark/actions/TestExpireSnapshotsAction.java | 7 +-
.../spark/actions/TestRemoveOrphanFilesAction.java | 19 +-
.../spark/actions/TestRewriteDataFilesAction.java | 84 +-
.../spark/actions/TestRewriteManifestsAction.java | 10 +-
.../TestRewritePositionDeleteFilesAction.java | 65 ++
.../spark/actions/TestSparkFileRewriter.java | 47 +-
.../apache/iceberg/spark/data/GenericsHelpers.java | 104 +-
.../org/apache/iceberg/spark/data/RandomData.java | 4 +-
.../org/apache/iceberg/spark/data/TestHelpers.java | 114 +--
.../iceberg/spark/data/TestSparkParquetReader.java | 4 +-
.../vectorized/TestParquetVectorizedReads.java | 7 +-
.../spark/functions/TestSparkFunctions.java | 9 +-
.../apache/iceberg/spark/source/LogMessage.java | 2 +-
.../iceberg/spark/source/TestBaseReader.java | 2 +-
.../spark/source/TestCompressionSettings.java | 10 +-
.../spark/source/TestDataFrameWriterV2.java | 8 +-
.../iceberg/spark/source/TestDataFrameWrites.java | 38 +-
.../spark/source/TestDataSourceOptions.java | 45 +-
.../iceberg/spark/source/TestFilteredScan.java | 4 +-
.../spark/source/TestForwardCompatibility.java | 6 +-
.../spark/source/TestIcebergSourceTablesBase.java | 9 +-
.../iceberg/spark/source/TestIcebergSpark.java | 9 +-
.../spark/source/TestIdentityPartitionData.java | 2 +-
.../TestMetadataTablesWithPartitionEvolution.java | 4 +-
.../iceberg/spark/source/TestPartitionPruning.java | 2 +-
.../iceberg/spark/source/TestPathIdentifier.java | 7 +-
.../iceberg/spark/source/TestReadProjection.java | 5 +-
.../TestRequiredDistributionAndOrdering.java | 5 +-
.../spark/source/TestSnapshotSelection.java | 23 +-
.../source/TestSparkCatalogCacheExpiration.java | 24 +-
.../iceberg/spark/source/TestSparkDataWrite.java | 7 +-
.../spark/source/TestSparkMetadataColumns.java | 6 +-
.../iceberg/spark/source/TestSparkReadMetrics.java | 98 +-
.../apache/iceberg/spark/source/TestSparkScan.java | 92 +-
.../spark/source/TestStructuredStreaming.java | 10 +-
.../spark/source/TestStructuredStreamingRead3.java | 40 +-
.../spark/source/TestWriteMetricsConfig.java | 5 +-
.../iceberg/spark/sql/TestAggregatePushDown.java | 67 +-
.../apache/iceberg/spark/sql/TestCreateTable.java | 23 +-
.../iceberg/spark/sql/TestCreateTableAsSelect.java | 4 +-
.../apache/iceberg/spark/sql/TestDeleteFrom.java | 6 +-
.../apache/iceberg/spark/sql/TestDropTable.java | 5 +-
.../iceberg/spark/sql/TestFilterPushDown.java | 10 +-
.../apache/iceberg/spark/sql/TestNamespaceSQL.java | 5 +-
.../sql/TestPartitionedWritesToWapBranch.java | 8 +-
.../org/apache/iceberg/spark/sql/TestSelect.java | 21 +-
.../iceberg/spark/sql/TestSparkBucketFunction.java | 61 +-
.../iceberg/spark/sql/TestSparkDaysFunction.java | 11 +-
.../iceberg/spark/sql/TestSparkHoursFunction.java | 11 +-
.../iceberg/spark/sql/TestSparkMonthsFunction.java | 16 +-
.../spark/sql/TestSparkTruncateFunction.java | 53 +-
.../iceberg/spark/sql/TestSparkYearsFunction.java | 16 +-
.../spark/sql/TestTimestampWithoutZone.java | 13 +-
.../spark/sql/TestUnpartitionedWritesToBranch.java | 5 +-
.../spark/sql/UnpartitionedWritesTestBase.java | 7 +-
spark/v3.5/build.gradle | 2 +-
.../SparkRowLevelOperationsTestBase.java | 2 +-
.../spark/extensions/TestAddFilesProcedure.java | 14 +-
.../spark/extensions/TestAlterTableSchema.java | 16 +-
.../spark/extensions/TestAncestorsOfProcedure.java | 10 +-
.../iceberg/spark/extensions/TestBranchDDL.java | 25 +-
.../spark/extensions/TestCallStatementParser.java | 4 +-
.../spark/extensions/TestChangelogTable.java | 5 +-
.../TestCherrypickSnapshotProcedure.java | 17 +-
.../spark/extensions/TestConflictValidation.java | 21 +-
.../spark/extensions/TestCopyOnWriteDelete.java | 4 +-
.../spark/extensions/TestCopyOnWriteMerge.java | 4 +-
.../spark/extensions/TestCopyOnWriteUpdate.java | 4 +-
.../iceberg/spark/extensions/TestDelete.java | 13 +-
.../extensions/TestExpireSnapshotsProcedure.java | 26 +-
.../apache/iceberg/spark/extensions/TestMerge.java | 94 +-
.../spark/extensions/TestMergeOnReadDelete.java | 5 +-
.../extensions/TestMigrateTableProcedure.java | 53 +-
.../extensions/TestPublishChangesProcedure.java | 13 +-
.../extensions/TestRemoveOrphanFilesProcedure.java | 31 +-
.../spark/extensions/TestReplaceBranch.java | 6 +-
.../TestRequiredDistributionAndOrdering.java | 5 +-
.../extensions/TestRewriteDataFilesProcedure.java | 35 +-
.../extensions/TestRewriteManifestsProcedure.java | 16 +-
.../TestRollbackToSnapshotProcedure.java | 23 +-
.../TestRollbackToTimestampProcedure.java | 19 +-
.../TestSetCurrentSnapshotProcedure.java | 29 +-
.../TestSetWriteDistributionAndOrdering.java | 4 +-
.../extensions/TestSnapshotTableProcedure.java | 51 +-
.../iceberg/spark/extensions/TestTagDDL.java | 28 +-
.../iceberg/spark/extensions/TestUpdate.java | 45 +-
.../iceberg/spark/extensions/TestWriteAborts.java | 5 +-
spark/v3.5/spark-runtime/NOTICE | 2 +-
.../java/org/apache/iceberg/spark/SmokeTest.java | 37 +-
.../iceberg/spark/action/RandomGeneratingUDF.java | 5 +-
.../iceberg/spark/PruneColumnsWithReordering.java | 2 +-
.../java/org/apache/iceberg/spark/Spark3Util.java | 2 +-
.../org/apache/iceberg/spark/SparkTableUtil.java | 141 ++-
.../actions/DeleteOrphanFilesSparkAction.java | 2 +-
.../spark/actions/MigrateTableSparkAction.java | 11 +-
.../spark/actions/RewriteDataFilesSparkAction.java | 2 +-
.../spark/actions/RewriteManifestsSparkAction.java | 28 +-
.../RewritePositionDeleteFilesSparkAction.java | 15 +-
.../spark/actions/SnapshotTableSparkAction.java | 11 +-
.../iceberg/spark/data/SparkParquetWriters.java | 2 +-
.../spark/procedures/MigrateTableProcedure.java | 10 +-
.../spark/procedures/SnapshotTableProcedure.java | 9 +-
.../spark/source/SparkFileWriterFactory.java | 2 +-
.../spark/source/SparkMicroBatchStream.java | 1 -
.../spark/source/SparkPositionDeltaWrite.java | 10 +-
.../iceberg/spark/source/SparkScanBuilder.java | 140 +--
.../spark/source/SparkStagedScanBuilder.java | 2 +-
.../apache/iceberg/spark/source/SparkWrite.java | 10 +-
.../analysis/NoSuchProcedureException.java | 2 +-
.../apache/iceberg/spark/SparkTestHelperBase.java | 17 +-
.../java/org/apache/iceberg/spark/TestBase.java | 2 +-
.../iceberg/spark/TestSparkCatalogOperations.java | 2 +-
.../iceberg/spark/actions/TestCreateActions.java | 2 +-
.../spark/actions/TestMigrateTableAction.java | 68 ++
.../spark/actions/TestRewriteDataFilesAction.java | 59 +-
.../TestRewritePositionDeleteFilesAction.java | 65 ++
.../spark/actions/TestSnapshotTableAction.java | 68 ++
.../spark/actions/TestSparkFileRewriter.java | 74 +-
.../org/apache/iceberg/spark/data/RandomData.java | 4 +-
.../spark/functions/TestSparkFunctions.java | 9 +-
.../apache/iceberg/spark/source/LogMessage.java | 2 +-
.../iceberg/spark/source/TestBaseReader.java | 2 +-
.../spark/source/TestCompressionSettings.java | 2 +-
.../iceberg/spark/source/TestDataFrameWrites.java | 33 +-
.../spark/source/TestDataSourceOptions.java | 32 +-
.../spark/source/TestIdentityPartitionData.java | 2 +-
.../iceberg/spark/source/TestPartitionPruning.java | 2 +-
.../spark/source/TestSparkMetadataColumns.java | 47 +
.../spark/source/TestStructuredStreaming.java | 6 +-
.../spark/source/TestStructuredStreamingRead3.java | 2 +-
.../iceberg/spark/sql/TestAggregatePushDown.java | 65 +-
.../apache/iceberg/spark/sql/TestCreateTable.java | 21 +-
.../apache/iceberg/spark/sql/TestDeleteFrom.java | 5 +-
.../apache/iceberg/spark/sql/TestDropTable.java | 4 +-
.../apache/iceberg/spark/sql/TestNamespaceSQL.java | 6 +-
.../sql/TestPartitionedWritesToWapBranch.java | 8 +-
.../iceberg/spark/sql/TestSparkBucketFunction.java | 59 +-
.../spark/sql/UnpartitionedWritesTestBase.java | 6 +-
867 files changed, 13390 insertions(+), 11569 deletions(-)
copy api/src/main/java/org/apache/iceberg/types/{AssignFreshIds.java => AssignIds.java} (66%)
delete mode 100644 api/src/test/java/org/apache/iceberg/AssertHelpers.java
create mode 100644 common/src/test/java/org/apache/iceberg/common/TestDynConstructors.java
create mode 100644 core/src/main/java/org/apache/iceberg/DataTaskParser.java
create mode 100644 core/src/main/java/org/apache/iceberg/ScanTaskParser.java
create mode 100644 core/src/test/java/org/apache/iceberg/TestDataTaskParser.java
copy api/src/test/java/org/apache/iceberg/metrics/TestDefaultCounter.java => core/src/test/java/org/apache/iceberg/TestScanTaskParser.java (51%)
copy flink/{v1.18/flink/src/test/java/org/apache/iceberg/flink/HadoopTableResource.java => v1.17/flink/src/test/java/org/apache/iceberg/flink/HadoopTableExtension.java} (72%)
copy flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/{HadoopTableResource.java => HadoopTableExtension.java} (72%)
copy flink/{v1.18/flink/src/test/java/org/apache/iceberg/flink/HadoopTableResource.java => v1.19/flink/src/test/java/org/apache/iceberg/flink/HadoopTableExtension.java} (72%)
create mode 100644 spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestMigrateTableAction.java
create mode 100644 spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/actions/TestSnapshotTableAction.java