You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by fo...@apache.org on 2023/05/01 21:43:50 UTC
[iceberg] branch fd-add-missing-last-column-id updated (07e8839b6c -> 10a1af9574)
This is an automated email from the ASF dual-hosted git repository.
fokko pushed a change to branch fd-add-missing-last-column-id
in repository https://gitbox.apache.org/repos/asf/iceberg.git
omit 07e8839b6c Make it in Java optional as well
omit 75343046d9 Update rest-catalog-open-api.yaml
omit e857d4153b Spec: Add missing `last-column-id`
add 8ca688531c Core: Move table-creation-without-namespace-test to CatalogTests (#7349)
add faddf308ed Spark: Refactor SparkReadConf to use primitive type (#7429)
add 449509df46 Spark 3.4: Remove deprecated classes (#7448)
add 97a99c3cf4 Arrow: Convert dict encoded vectors to their expected Arrow vector types (#3024)
add 1969775f14 Spark: Fixed Typo in Spark Read Option Vectorization Javadoc (#7439)
add 91327e7c57 Spark 3.4: Remove no longer needed write extensions (#7443)
add b1d25db291 Delta: Add version and timestamp tags for each Delta Lake transaction (#7450)
add 5f156d73bd Delta: Fix snapshotDataFilesCount and use Immutable for result (#7454)
add 849ceac48c updates to remove mkdocs warnings from_bytes()
add 2edebcc633 updated comments for mkdocs WARNINGS to be removed
add ea416b25e6 Python: fix `mkdocs` warnings from `conversions.py` (#7462)
add 3d651a1fc2 Spark 3.4: Support rate limit in Spark Streaming (#7422)
add 166fde6d3f Merge branch 'apache:master' into master
add 99f96199f6 removal of warnings from mkdocs serve --strict
add 78aafdb55e Python: fix mkdocs warnings from visitors.py (#7464)
add 4a1b307972 Merge branch 'apache:master' into master
add b0e16a98e7 changes to SortOrder init() added order_id arg
add 6f3fe3c030 reverted...
add 27f9c16c3d updated for keyword args vs args order_id
add 086162e697 removed blank line 123 fom sorting.py
add 58a2689a62 finalizing comments
add d3abd8d43b Remove spaces
add 8c5605dfd1 Python: fix mkdocs warnings from sorting.py (#7467)
add 1be7bcf4e4 Merge branch 'apache:master' into master
add 47776d23e9 fixed all mkdocs serve --strict warnings
add 74c4820a07 fixed trailing whitespace
add b83f0303d2 updates to comments for mkdocs serve --strict mode
add b04521ab89 updates to io/__init__.py for mkdocs warns removal
add 91282c4d43 updates to remove mkdocs warns from types.py
add de2624fa8e updates for schema.py to remove mkdocs warns
add c2c4a38bb9 updates to remove mkdoc warns from literals.py
add 5fcdc596a0 final mkdocs fixed transforms.py
add 1092d4be93 request PR changes from code review
add 4ae09c08df reuest format changes for comments
add 251c9fa796 Spark: Fix failing Spark streaming rate limit unit test (#7470)
add c8b80b0b31 Build: Bump ray from 2.3.1 to 2.4.0 in /python (#7479)
add dc82f98f01 Build: Bump mkdocs-gen-files from 0.4.0 to 0.5.0 in /python (#7478)
add 4d54f4cf8f Build: Bump mkdocstrings-python from 0.8.3 to 0.9.0 in /python (#7475)
add 9414adc21d Build: Bump pyyaml from 5.4.1 to 6.0 in /python (#7477)
add 5e6a0147d8 Build: Bump mkdocs-material from 9.1.5 to 9.1.8 in /python (#7476)
add 4a4b604f83 Merge branch 'apache:master' into master
add a65b707125 fix pymdownx markdown_ext for code hihglights
add 9fb90acdac Python: Fix warnings in documentation (#7472)
add 725e66aacb Merge branch 'apache:master' into master
add 9de9a2ae77 Spark 3.4: Switch to built-in DELETE implementation (#7453)
add 2b8850b619 Python: Fix code highlighting in docs (#7484)
add a0b1df2866 Spark: Remove Usage of deprecated AssertHelpers in spark-sql (#7486)
add f8b1601902 Spark 3.3: Remove deprecated FileScanTaskSetManager (#7489)
add 4ce5318e5c Python: Update pre-commit to the latest version (#7436)
add bca9d35837 Merge branch 'master' of github.com:apache/iceberg into fd-add-missing-last-column-id
add 10a1af9574 Add test
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (07e8839b6c)
\
N -- N -- N refs/heads/fd-add-missing-last-column-id (10a1af9574)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/python-ci-docs.yml | 2 +-
.../iceberg/arrow/DictEncodedArrowConverter.java | 225 +++++++++++++++++
.../iceberg/arrow/vectorized/ArrowReader.java | 3 +-
.../arrow/vectorized/ArrowVectorAccessors.java | 22 +-
.../iceberg/arrow/vectorized/ColumnVector.java | 18 +-
.../iceberg/arrow/vectorized/ColumnarBatch.java | 2 +-
.../iceberg/arrow/vectorized/VectorHolder.java | 26 +-
.../arrow/vectorized/VectorizedArrowReader.java | 4 +-
.../iceberg/arrow/vectorized/ArrowReaderTest.java | 122 +++++-----
build.gradle | 2 +
.../org/apache/iceberg/MetadataUpdateParser.java | 7 +-
.../apache/iceberg/inmemory/InMemoryCatalog.java | 14 +-
.../apache/iceberg/TestMetadataUpdateParser.java | 11 +
.../org/apache/iceberg/catalog/CatalogTests.java | 12 +
.../iceberg/inmemory/TestInMemoryCatalog.java | 17 --
.../iceberg/delta/TestSnapshotDeltaLakeTable.java | 105 ++++++--
.../delta/BaseSnapshotDeltaLakeTableAction.java | 58 +++--
.../BaseSnapshotDeltaLakeTableActionResult.java | 33 ---
.../iceberg/delta/SnapshotDeltaLakeTable.java | 3 +
open-api/rest-catalog-open-api.yaml | 3 -
python/.pre-commit-config.yaml | 15 +-
python/mkdocs/mkdocs.yml | 3 +
python/mkdocs/requirements.txt | 6 +-
python/poetry.lock | 209 +++++++---------
python/pyiceberg/conversions.py | 16 +-
python/pyiceberg/expressions/__init__.py | 2 +-
python/pyiceberg/expressions/literals.py | 17 +-
python/pyiceberg/expressions/visitors.py | 38 +--
python/pyiceberg/io/__init__.py | 24 +-
python/pyiceberg/io/fsspec.py | 20 +-
python/pyiceberg/io/pyarrow.py | 23 +-
python/pyiceberg/schema.py | 20 +-
python/pyiceberg/table/sorting.py | 4 +-
python/pyiceberg/transforms.py | 6 +-
python/pyiceberg/typedef.py | 2 +-
python/pyiceberg/types.py | 12 +-
python/pyiceberg/utils/decimal.py | 2 +-
python/pyproject.toml | 2 +-
python/tests/test_transforms.py | 8 +-
python/tests/utils/test_bin_packing.py | 2 +-
.../org/apache/iceberg/spark/SparkReadConf.java | 2 +-
.../org/apache/iceberg/spark/SparkReadOptions.java | 2 +-
.../spark/source/SparkMicroBatchStream.java | 8 +-
.../org/apache/iceberg/spark/SparkReadConf.java | 2 +-
.../org/apache/iceberg/spark/SparkReadOptions.java | 2 +-
.../spark/source/SparkMicroBatchStream.java | 8 +-
.../iceberg/spark/FileScanTaskSetManager.java | 79 ------
.../org/apache/iceberg/spark/SparkReadConf.java | 6 +-
.../org/apache/iceberg/spark/SparkReadOptions.java | 2 +-
.../spark/source/SparkMicroBatchStream.java | 12 +-
.../extensions/IcebergSparkSessionExtensions.scala | 5 -
.../analysis/RewriteDeleteFromIcebergTable.scala | 118 ---------
.../catalyst/analysis/RewriteMergeIntoTable.scala | 4 +-
.../analysis/RewriteRowLevelIcebergCommand.scala | 4 +-
.../expressions/ExtendedV2ExpressionUtils.scala | 103 --------
.../ExtendedReplaceNullWithFalseInPredicate.scala | 4 -
.../ExtendedSimplifyConditionalsInPredicate.scala | 3 -
.../IcebergSparkSqlExtensionsParser.scala | 5 -
.../plans/logical/DeleteFromIcebergTable.scala | 49 ----
.../catalyst/plans/logical/WriteIcebergDelta.scala | 6 +-
.../connector/expressions/TruncateTransform.scala | 38 ---
.../v2/ExtendedDataSourceV2Strategy.scala | 20 --
.../v2/ExtendedDistributionAndOrderingUtils.scala | 85 -------
.../datasources/v2/ExtendedV2Writes.scala | 66 +----
...ptimizeMetadataOnlyDeleteFromIcebergTable.scala | 73 ------
.../RowLevelCommandDynamicPruning.scala | 10 +-
.../iceberg/spark/extensions/TestDelete.java | 15 +-
.../spark/extensions/TestIcebergExpressions.java | 74 ------
.../extensions/TestRewriteDataFilesProcedure.java | 8 +
.../java/org/apache/iceberg/spark/BaseCatalog.java | 37 +--
.../iceberg/spark/FileRewriteCoordinator.java | 14 --
.../iceberg/spark/FileScanTaskSetManager.java | 79 ------
.../org/apache/iceberg/spark/SortOrderToSpark.java | 2 +-
.../java/org/apache/iceberg/spark/Spark3Util.java | 17 --
.../iceberg/spark/SparkCachedTableCatalog.java | 2 +-
.../iceberg/spark/SparkFunctionCatalog.java} | 33 ++-
.../org/apache/iceberg/spark/SparkReadConf.java | 18 +-
.../org/apache/iceberg/spark/SparkReadOptions.java | 9 +-
.../org/apache/iceberg/spark/SparkTableUtil.java | 51 +---
.../java/org/apache/iceberg/spark/SparkUtil.java | 22 --
.../apache/iceberg/spark/SupportsFunctions.java | 63 +++++
.../spark/actions/SparkBinPackStrategy.java | 103 --------
.../spark/actions/SparkShufflingDataRewriter.java | 83 +++++--
.../spark/actions/SparkSizeBasedDataRewriter.java | 2 +-
.../spark/actions/SparkSortDataRewriter.java | 12 +-
.../iceberg/spark/actions/SparkSortStrategy.java | 187 --------------
.../spark/actions/SparkZOrderDataRewriter.java | 11 +-
.../iceberg/spark/actions/SparkZOrderStrategy.java | 268 ---------------------
.../vectorized/VectorizedSparkParquetReaders.java | 24 --
.../spark/source/SparkMicroBatchStream.java | 195 +++++++++++++--
.../iceberg/spark/source/SparkWriteBuilder.java | 12 +-
.../expressions/TransformExpressions.scala | 160 ------------
.../utils/DistributionAndOrderingUtils.scala | 189 ---------------
.../iceberg/spark/TestFileRewriteCoordinator.java | 6 +-
.../spark/actions/TestRemoveOrphanFilesAction.java | 8 +-
.../spark/actions/TestRewriteDataFilesAction.java | 10 +
.../data/TestSparkParquetReadMetadataColumns.java | 6 +-
.../TestRequiredDistributionAndOrdering.java | 17 +-
.../spark/source/TestStructuredStreamingRead3.java | 84 ++++++-
.../apache/iceberg/spark/sql/TestAlterTable.java | 37 ++-
.../apache/iceberg/spark/sql/TestCreateTable.java | 27 +--
.../apache/iceberg/spark/sql/TestDeleteFrom.java | 11 +-
.../apache/iceberg/spark/sql/TestDropTable.java | 11 +-
.../apache/iceberg/spark/sql/TestNamespaceSQL.java | 10 +-
.../org/apache/iceberg/spark/sql/TestSelect.java | 99 ++++----
.../iceberg/spark/sql/TestSparkBucketFunction.java | 166 ++++++-------
.../iceberg/spark/sql/TestSparkDaysFunction.java | 38 ++-
.../iceberg/spark/sql/TestSparkHoursFunction.java | 39 ++-
.../iceberg/spark/sql/TestSparkMonthsFunction.java | 38 ++-
.../spark/sql/TestSparkTruncateFunction.java | 160 ++++++------
.../iceberg/spark/sql/TestSparkYearsFunction.java | 38 ++-
.../spark/sql/TestTimestampWithoutZone.java | 14 +-
.../spark/sql/UnpartitionedWritesTestBase.java | 29 +--
113 files changed, 1555 insertions(+), 2819 deletions(-)
create mode 100644 arrow/src/main/java/org/apache/iceberg/arrow/DictEncodedArrowConverter.java
delete mode 100644 delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableActionResult.java
delete mode 100644 spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/FileScanTaskSetManager.java
delete mode 100644 spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteDeleteFromIcebergTable.scala
delete mode 100644 spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExtendedV2ExpressionUtils.scala
delete mode 100644 spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DeleteFromIcebergTable.scala
delete mode 100644 spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/connector/expressions/TruncateTransform.scala
delete mode 100644 spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDistributionAndOrderingUtils.scala
delete mode 100644 spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/OptimizeMetadataOnlyDeleteFromIcebergTable.scala
delete mode 100644 spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestIcebergExpressions.java
delete mode 100644 spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/FileScanTaskSetManager.java
copy spark/{v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkMetadataColumn.java => v3.4/spark/src/main/java/org/apache/iceberg/spark/SparkFunctionCatalog.java} (59%)
create mode 100644 spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/SupportsFunctions.java
delete mode 100644 spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/actions/SparkBinPackStrategy.java
delete mode 100644 spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/actions/SparkSortStrategy.java
delete mode 100644 spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/actions/SparkZOrderStrategy.java
delete mode 100644 spark/v3.4/spark/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpressions.scala
delete mode 100644 spark/v3.4/spark/src/main/scala/org/apache/spark/sql/catalyst/utils/DistributionAndOrderingUtils.scala