You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/02/14 17:37:45 UTC
[GitHub] [iceberg] Zhangg7723 opened a new pull request #4119: [update] modify the configuration.md
Zhangg7723 opened a new pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119
Modify configuration.md doc with read and write properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Zhangg7723 commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
Zhangg7723 commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808602959
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -34,29 +34,50 @@ Iceberg tables support table properties to configure table behavior, like the de
| read.split.metadata-target-size | 33554432 (32 MB) | Target size when combining metadata input splits |
| read.split.planning-lookback | 10 | Number of bins to consider when combining input splits |
| read.split.open-file-cost | 4194304 (4 MB) | The estimated cost to open a file, used as a minimum weight when combining splits. |
+| read.parquet.vectorization.enabled| false | Enables parquet vectorization read |
+| read.parquet.vectorization.batch-size| 5000 | The batch size of parquet vectorization read |
+| read.orc.vectorization.enabled | false | Enables orc vectorization read |
+| read.orc.vectorization.batch-size | 5000 | The batch size of orc vectorization read |
### Write properties
| Property | Default | Description |
| ---------------------------------- | ------------------ | -------------------------------------------------- |
| write.format.default | parquet | Default file format for the table; parquet, avro, or orc |
+| write.delete.format.default | | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), gzip, snappy, uncompressed |
+| write.avro.compression-level | null | Avro compression level |
| write.location-provider.impl | null | Optional custom implemention for LocationProvider |
| write.metadata.compression-codec | none | Metadata compression codec; none or gzip |
| write.metadata.metrics.default | truncate(16) | Default metrics mode for all columns in the table; none, counts, truncate(length), or full |
| write.metadata.metrics.column.col1 | (not set) | Metrics mode for column 'col1' to allow per-column tuning; none, counts, truncate(length), or full |
| write.target-file-size-bytes | 536870912 (512 MB) | Controls the size of files generated to target about this many bytes |
+| write.delete.target-file-size-bytes| 67108864 (64 MB) | Controls the size of delete files generated to target about this many bytes |
| write.distribution-mode | none | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder |
+| write.delete.distribution-mode | hash | Defines distribution of write delete data |
| write.wap.enabled | false | Enables write-audit-publish writes |
| write.summary.partition-limit | 0 | Includes partition-level summary stats in snapshot summaries if the changed partition count is less than this limit |
| write.metadata.delete-after-commit.enabled | false | Controls whether to delete the oldest version metadata files after commit |
| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
| write.spark.fanout.enabled | false | Enables Partitioned-Fanout-Writer writes in Spark |
+| write.object-storage.enabled | false | Enables object storage write like s3 |
+| write.data.path | a "data" folder underneath the root path of the table | Defines the path of data files |
+| write.metadata.path | a "metadata" folder underneath the root path of the table | Defines the path of metadata files |
+| write.manifest-lists.enabled | true | Enables to list manifest |
+| write.delete.isolation-level | serializable | Defines the isolation level of write delete |
+| write.delete.mode | copy-on-write | Defines the write delete mode |
+| write.update.isolation-level | serializable | Defines the isolation level of write update |
+| write.update.mode | copy-on-write | Defines the write update mode |
+| write.merge.isolation-level | serializable | Defines the isolation level of write merge |
+| write.merge.mode | copy-on-write | Defines the write merge mode |
+| write.upsert.enabled | false | Enables the upsert writes |
+
+
Review comment:
@rdblue Thanks for all your earnest reviews，I will double check my commit。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808516888
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -102,6 +122,9 @@ Iceberg catalogs support using catalog properties to configure catalog behaviors
| warehouse | null | the root path of the data warehouse |
| uri | null | a URI string, such as Hive metastore URI |
| clients | 2 | client pool size |
+| cache-enabled | true | Controls whether the catalog will cache table entries upon load. |
+| cache.expiration-interval-ms | 30000 | Controls the duration for which entries in the catalog are cached. |
+| client.pool.cache.eviction-interval-ms | 5000 | Controls the duration for which clients in the pool |
Review comment:
I don't understand this description.
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -102,6 +122,9 @@ Iceberg catalogs support using catalog properties to configure catalog behaviors
| warehouse | null | the root path of the data warehouse |
| uri | null | a URI string, such as Hive metastore URI |
| clients | 2 | client pool size |
+| cache-enabled | true | Controls whether the catalog will cache table entries upon load. |
Review comment:
No need for punctuation at the end of descriptions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808513891
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -34,29 +34,50 @@ Iceberg tables support table properties to configure table behavior, like the de
| read.split.metadata-target-size | 33554432 (32 MB) | Target size when combining metadata input splits |
| read.split.planning-lookback | 10 | Number of bins to consider when combining input splits |
| read.split.open-file-cost | 4194304 (4 MB) | The estimated cost to open a file, used as a minimum weight when combining splits. |
+| read.parquet.vectorization.enabled| false | Enables parquet vectorization read |
+| read.parquet.vectorization.batch-size| 5000 | The batch size of parquet vectorization read |
+| read.orc.vectorization.enabled | false | Enables orc vectorization read |
+| read.orc.vectorization.batch-size | 5000 | The batch size of orc vectorization read |
### Write properties
| Property | Default | Description |
| ---------------------------------- | ------------------ | -------------------------------------------------- |
| write.format.default | parquet | Default file format for the table; parquet, avro, or orc |
+| write.delete.format.default | | Default delete file format for the table; parquet, avro, or orc |
Review comment:
Looks like formatting is misaligned in a few places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808513756
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -34,29 +34,50 @@ Iceberg tables support table properties to configure table behavior, like the de
| read.split.metadata-target-size | 33554432 (32 MB) | Target size when combining metadata input splits |
| read.split.planning-lookback | 10 | Number of bins to consider when combining input splits |
| read.split.open-file-cost | 4194304 (4 MB) | The estimated cost to open a file, used as a minimum weight when combining splits. |
+| read.parquet.vectorization.enabled| false | Enables parquet vectorization read |
+| read.parquet.vectorization.batch-size| 5000 | The batch size of parquet vectorization read |
+| read.orc.vectorization.enabled | false | Enables orc vectorization read |
+| read.orc.vectorization.batch-size | 5000 | The batch size of orc vectorization read |
### Write properties
| Property | Default | Description |
| ---------------------------------- | ------------------ | -------------------------------------------------- |
| write.format.default | parquet | Default file format for the table; parquet, avro, or orc |
+| write.delete.format.default | | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), gzip, snappy, uncompressed |
+| write.avro.compression-level | null | Avro compression level |
| write.location-provider.impl | null | Optional custom implemention for LocationProvider |
| write.metadata.compression-codec | none | Metadata compression codec; none or gzip |
| write.metadata.metrics.default | truncate(16) | Default metrics mode for all columns in the table; none, counts, truncate(length), or full |
| write.metadata.metrics.column.col1 | (not set) | Metrics mode for column 'col1' to allow per-column tuning; none, counts, truncate(length), or full |
| write.target-file-size-bytes | 536870912 (512 MB) | Controls the size of files generated to target about this many bytes |
+| write.delete.target-file-size-bytes| 67108864 (64 MB) | Controls the size of delete files generated to target about this many bytes |
| write.distribution-mode | none | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder |
+| write.delete.distribution-mode | hash | Defines distribution of write delete data |
| write.wap.enabled | false | Enables write-audit-publish writes |
| write.summary.partition-limit | 0 | Includes partition-level summary stats in snapshot summaries if the changed partition count is less than this limit |
| write.metadata.delete-after-commit.enabled | false | Controls whether to delete the oldest version metadata files after commit |
| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
| write.spark.fanout.enabled | false | Enables Partitioned-Fanout-Writer writes in Spark |
+| write.object-storage.enabled | false | Enables object storage write like s3 |
+| write.data.path | a "data" folder underneath the root path of the table | Defines the path of data files |
+| write.metadata.path | a "metadata" folder underneath the root path of the table | Defines the path of metadata files |
+| write.manifest-lists.enabled | true | Enables to list manifest |
+| write.delete.isolation-level | serializable | Defines the isolation level of write delete |
+| write.delete.mode | copy-on-write | Defines the write delete mode |
+| write.update.isolation-level | serializable | Defines the isolation level of write update |
+| write.update.mode | copy-on-write | Defines the write update mode |
+| write.merge.isolation-level | serializable | Defines the isolation level of write merge |
+| write.merge.mode | copy-on-write | Defines the write merge mode |
+| write.upsert.enabled | false | Enables the upsert writes |
+
+
Review comment:
Extra newlines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r806338624
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -75,7 +96,6 @@ Iceberg tables support table properties to configure table behavior, like the de
| commit.manifest-merge.enabled | true | Controls whether to automatically merge manifests on writes |
| history.expire.max-snapshot-age-ms | 432000000 (5 days) | Default max age of snapshots to keep while expiring snapshots |
| history.expire.min-snapshots-to-keep | 1 | Default min number of snapshots to keep while expiring snapshots |
-| history.expire.max-ref-age-ms | `Long.MAX_VALUE` (forever) | For snapshot references except the `main` branch, default max age of snapshot references to keep while expiring snapshots. The `main` branch never expires. |
Review comment:
should not be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808514439
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -34,29 +34,50 @@ Iceberg tables support table properties to configure table behavior, like the de
| read.split.metadata-target-size | 33554432 (32 MB) | Target size when combining metadata input splits |
| read.split.planning-lookback | 10 | Number of bins to consider when combining input splits |
| read.split.open-file-cost | 4194304 (4 MB) | The estimated cost to open a file, used as a minimum weight when combining splits. |
+| read.parquet.vectorization.enabled| false | Enables parquet vectorization read |
+| read.parquet.vectorization.batch-size| 5000 | The batch size of parquet vectorization read |
+| read.orc.vectorization.enabled | false | Enables orc vectorization read |
+| read.orc.vectorization.batch-size | 5000 | The batch size of orc vectorization read |
### Write properties
| Property | Default | Description |
| ---------------------------------- | ------------------ | -------------------------------------------------- |
| write.format.default | parquet | Default file format for the table; parquet, avro, or orc |
+| write.delete.format.default | | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), gzip, snappy, uncompressed |
+| write.avro.compression-level | null | Avro compression level |
| write.location-provider.impl | null | Optional custom implemention for LocationProvider |
| write.metadata.compression-codec | none | Metadata compression codec; none or gzip |
| write.metadata.metrics.default | truncate(16) | Default metrics mode for all columns in the table; none, counts, truncate(length), or full |
| write.metadata.metrics.column.col1 | (not set) | Metrics mode for column 'col1' to allow per-column tuning; none, counts, truncate(length), or full |
| write.target-file-size-bytes | 536870912 (512 MB) | Controls the size of files generated to target about this many bytes |
+| write.delete.target-file-size-bytes| 67108864 (64 MB) | Controls the size of delete files generated to target about this many bytes |
| write.distribution-mode | none | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder |
+| write.delete.distribution-mode | hash | Defines distribution of write delete data |
| write.wap.enabled | false | Enables write-audit-publish writes |
| write.summary.partition-limit | 0 | Includes partition-level summary stats in snapshot summaries if the changed partition count is less than this limit |
| write.metadata.delete-after-commit.enabled | false | Controls whether to delete the oldest version metadata files after commit |
| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
| write.spark.fanout.enabled | false | Enables Partitioned-Fanout-Writer writes in Spark |
+| write.object-storage.enabled | false | Enables object storage write like s3 |
+| write.data.path | a "data" folder underneath the root path of the table | Defines the path of data files |
+| write.metadata.path | a "metadata" folder underneath the root path of the table | Defines the path of metadata files |
+| write.manifest-lists.enabled | true | Enables to list manifest |
Review comment:
This should be removed. This is a compatibility property and not something people should use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Zhangg7723 commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
Zhangg7723 commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r806385967
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -75,7 +96,6 @@ Iceberg tables support table properties to configure table behavior, like the de
| commit.manifest-merge.enabled | true | Controls whether to automatically merge manifests on writes |
| history.expire.max-snapshot-age-ms | 432000000 (5 days) | Default max age of snapshots to keep while expiring snapshots |
| history.expire.min-snapshots-to-keep | 1 | Default min number of snapshots to keep while expiring snapshots |
-| history.expire.max-ref-age-ms | `Long.MAX_VALUE` (forever) | For snapshot references except the `main` branch, default max age of snapshot references to keep while expiring snapshots. The `main` branch never expires. |
Review comment:
but I didn't find any usage of this parameter，is it deprecated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r806338624
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -75,7 +96,6 @@ Iceberg tables support table properties to configure table behavior, like the de
| commit.manifest-merge.enabled | true | Controls whether to automatically merge manifests on writes |
| history.expire.max-snapshot-age-ms | 432000000 (5 days) | Default max age of snapshots to keep while expiring snapshots |
| history.expire.min-snapshots-to-keep | 1 | Default min number of snapshots to keep while expiring snapshots |
-| history.expire.max-ref-age-ms | `Long.MAX_VALUE` (forever) | For snapshot references except the `main` branch, default max age of snapshot references to keep while expiring snapshots. The `main` branch never expires. |
Review comment:
should not be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808515712
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -34,29 +34,50 @@ Iceberg tables support table properties to configure table behavior, like the de
| read.split.metadata-target-size | 33554432 (32 MB) | Target size when combining metadata input splits |
| read.split.planning-lookback | 10 | Number of bins to consider when combining input splits |
| read.split.open-file-cost | 4194304 (4 MB) | The estimated cost to open a file, used as a minimum weight when combining splits. |
+| read.parquet.vectorization.enabled| false | Enables parquet vectorization read |
+| read.parquet.vectorization.batch-size| 5000 | The batch size of parquet vectorization read |
+| read.orc.vectorization.enabled | false | Enables orc vectorization read |
+| read.orc.vectorization.batch-size | 5000 | The batch size of orc vectorization read |
### Write properties
| Property | Default | Description |
| ---------------------------------- | ------------------ | -------------------------------------------------- |
| write.format.default | parquet | Default file format for the table; parquet, avro, or orc |
+| write.delete.format.default | | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), gzip, snappy, uncompressed |
+| write.avro.compression-level | null | Avro compression level |
| write.location-provider.impl | null | Optional custom implemention for LocationProvider |
| write.metadata.compression-codec | none | Metadata compression codec; none or gzip |
| write.metadata.metrics.default | truncate(16) | Default metrics mode for all columns in the table; none, counts, truncate(length), or full |
| write.metadata.metrics.column.col1 | (not set) | Metrics mode for column 'col1' to allow per-column tuning; none, counts, truncate(length), or full |
| write.target-file-size-bytes | 536870912 (512 MB) | Controls the size of files generated to target about this many bytes |
+| write.delete.target-file-size-bytes| 67108864 (64 MB) | Controls the size of delete files generated to target about this many bytes |
| write.distribution-mode | none | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder |
+| write.delete.distribution-mode | hash | Defines distribution of write delete data |
| write.wap.enabled | false | Enables write-audit-publish writes |
| write.summary.partition-limit | 0 | Includes partition-level summary stats in snapshot summaries if the changed partition count is less than this limit |
| write.metadata.delete-after-commit.enabled | false | Controls whether to delete the oldest version metadata files after commit |
| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
| write.spark.fanout.enabled | false | Enables Partitioned-Fanout-Writer writes in Spark |
+| write.object-storage.enabled | false | Enables object storage write like s3 |
+| write.data.path | a "data" folder underneath the root path of the table | Defines the path of data files |
+| write.metadata.path | a "metadata" folder underneath the root path of the table | Defines the path of metadata files |
+| write.manifest-lists.enabled | true | Enables to list manifest |
+| write.delete.isolation-level | serializable | Defines the isolation level of write delete |
+| write.delete.mode | copy-on-write | Defines the write delete mode |
Review comment:
The mode properties should state what the two modes are (copy-on-write and merge-on-read) and should state when merge-on-read can be used (v2 tables only)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808517260
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -102,6 +122,9 @@ Iceberg catalogs support using catalog properties to configure catalog behaviors
| warehouse | null | the root path of the data warehouse |
| uri | null | a URI string, such as Hive metastore URI |
| clients | 2 | client pool size |
+| cache-enabled | true | Controls whether the catalog will cache table entries upon load. |
+| cache.expiration-interval-ms | 30000 | Controls the duration for which entries in the catalog are cached. |
Review comment:
I think "entries" should be "tables".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808515403
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -34,29 +34,50 @@ Iceberg tables support table properties to configure table behavior, like the de
| read.split.metadata-target-size | 33554432 (32 MB) | Target size when combining metadata input splits |
| read.split.planning-lookback | 10 | Number of bins to consider when combining input splits |
| read.split.open-file-cost | 4194304 (4 MB) | The estimated cost to open a file, used as a minimum weight when combining splits. |
+| read.parquet.vectorization.enabled| false | Enables parquet vectorization read |
+| read.parquet.vectorization.batch-size| 5000 | The batch size of parquet vectorization read |
+| read.orc.vectorization.enabled | false | Enables orc vectorization read |
+| read.orc.vectorization.batch-size | 5000 | The batch size of orc vectorization read |
### Write properties
| Property | Default | Description |
| ---------------------------------- | ------------------ | -------------------------------------------------- |
| write.format.default | parquet | Default file format for the table; parquet, avro, or orc |
+| write.delete.format.default | | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), gzip, snappy, uncompressed |
+| write.avro.compression-level | null | Avro compression level |
| write.location-provider.impl | null | Optional custom implemention for LocationProvider |
| write.metadata.compression-codec | none | Metadata compression codec; none or gzip |
| write.metadata.metrics.default | truncate(16) | Default metrics mode for all columns in the table; none, counts, truncate(length), or full |
| write.metadata.metrics.column.col1 | (not set) | Metrics mode for column 'col1' to allow per-column tuning; none, counts, truncate(length), or full |
| write.target-file-size-bytes | 536870912 (512 MB) | Controls the size of files generated to target about this many bytes |
+| write.delete.target-file-size-bytes| 67108864 (64 MB) | Controls the size of delete files generated to target about this many bytes |
| write.distribution-mode | none | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder |
+| write.delete.distribution-mode | hash | Defines distribution of write delete data |
| write.wap.enabled | false | Enables write-audit-publish writes |
| write.summary.partition-limit | 0 | Includes partition-level summary stats in snapshot summaries if the changed partition count is less than this limit |
| write.metadata.delete-after-commit.enabled | false | Controls whether to delete the oldest version metadata files after commit |
| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
| write.spark.fanout.enabled | false | Enables Partitioned-Fanout-Writer writes in Spark |
+| write.object-storage.enabled | false | Enables object storage write like s3 |
+| write.data.path | a "data" folder underneath the root path of the table | Defines the path of data files |
+| write.metadata.path | a "metadata" folder underneath the root path of the table | Defines the path of metadata files |
+| write.manifest-lists.enabled | true | Enables to list manifest |
+| write.delete.isolation-level | serializable | Defines the isolation level of write delete |
Review comment:
The isolation level properties should define the different levels if you want to include them here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808514183
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -75,7 +96,6 @@ Iceberg tables support table properties to configure table behavior, like the de
| commit.manifest-merge.enabled | true | Controls whether to automatically merge manifests on writes |
| history.expire.max-snapshot-age-ms | 432000000 (5 days) | Default max age of snapshots to keep while expiring snapshots |
| history.expire.min-snapshots-to-keep | 1 | Default min number of snapshots to keep while expiring snapshots |
-| history.expire.max-ref-age-ms | `Long.MAX_VALUE` (forever) | For snapshot references except the `main` branch, default max age of snapshot references to keep while expiring snapshots. The `main` branch never expires. |
Review comment:
No, it isn't deprecated. We're in the process of adding it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Zhangg7723 commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
Zhangg7723 commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r806385967
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -75,7 +96,6 @@ Iceberg tables support table properties to configure table behavior, like the de
| commit.manifest-merge.enabled | true | Controls whether to automatically merge manifests on writes |
| history.expire.max-snapshot-age-ms | 432000000 (5 days) | Default max age of snapshots to keep while expiring snapshots |
| history.expire.min-snapshots-to-keep | 1 | Default min number of snapshots to keep while expiring snapshots |
-| history.expire.max-ref-age-ms | `Long.MAX_VALUE` (forever) | For snapshot references except the `main` branch, default max age of snapshot references to keep while expiring snapshots. The `main` branch never expires. |
Review comment:
but I didn't find any usage of this parameter，is it deprecated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808516208
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -34,29 +34,50 @@ Iceberg tables support table properties to configure table behavior, like the de
| read.split.metadata-target-size | 33554432 (32 MB) | Target size when combining metadata input splits |
| read.split.planning-lookback | 10 | Number of bins to consider when combining input splits |
| read.split.open-file-cost | 4194304 (4 MB) | The estimated cost to open a file, used as a minimum weight when combining splits. |
+| read.parquet.vectorization.enabled| false | Enables parquet vectorization read |
+| read.parquet.vectorization.batch-size| 5000 | The batch size of parquet vectorization read |
+| read.orc.vectorization.enabled | false | Enables orc vectorization read |
+| read.orc.vectorization.batch-size | 5000 | The batch size of orc vectorization read |
### Write properties
| Property | Default | Description |
| ---------------------------------- | ------------------ | -------------------------------------------------- |
| write.format.default | parquet | Default file format for the table; parquet, avro, or orc |
+| write.delete.format.default | | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), gzip, snappy, uncompressed |
+| write.avro.compression-level | null | Avro compression level |
| write.location-provider.impl | null | Optional custom implemention for LocationProvider |
| write.metadata.compression-codec | none | Metadata compression codec; none or gzip |
| write.metadata.metrics.default | truncate(16) | Default metrics mode for all columns in the table; none, counts, truncate(length), or full |
| write.metadata.metrics.column.col1 | (not set) | Metrics mode for column 'col1' to allow per-column tuning; none, counts, truncate(length), or full |
| write.target-file-size-bytes | 536870912 (512 MB) | Controls the size of files generated to target about this many bytes |
+| write.delete.target-file-size-bytes| 67108864 (64 MB) | Controls the size of delete files generated to target about this many bytes |
| write.distribution-mode | none | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder |
+| write.delete.distribution-mode | hash | Defines distribution of write delete data |
| write.wap.enabled | false | Enables write-audit-publish writes |
| write.summary.partition-limit | 0 | Includes partition-level summary stats in snapshot summaries if the changed partition count is less than this limit |
| write.metadata.delete-after-commit.enabled | false | Controls whether to delete the oldest version metadata files after commit |
| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
| write.spark.fanout.enabled | false | Enables Partitioned-Fanout-Writer writes in Spark |
+| write.object-storage.enabled | false | Enables object storage write like s3 |
+| write.data.path | a "data" folder underneath the root path of the table | Defines the path of data files |
+| write.metadata.path | a "metadata" folder underneath the root path of the table | Defines the path of metadata files |
+| write.manifest-lists.enabled | true | Enables to list manifest |
+| write.delete.isolation-level | serializable | Defines the isolation level of write delete |
+| write.delete.mode | copy-on-write | Defines the write delete mode |
+| write.update.isolation-level | serializable | Defines the isolation level of write update |
+| write.update.mode | copy-on-write | Defines the write update mode |
+| write.merge.isolation-level | serializable | Defines the isolation level of write merge |
+| write.merge.mode | copy-on-write | Defines the write merge mode |
+| write.upsert.enabled | false | Enables the upsert writes |
Review comment:
This is a Flink property, so it shouldn't go in this table. Upserts are not generally supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808515181
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -34,29 +34,50 @@ Iceberg tables support table properties to configure table behavior, like the de
| read.split.metadata-target-size | 33554432 (32 MB) | Target size when combining metadata input splits |
| read.split.planning-lookback | 10 | Number of bins to consider when combining input splits |
| read.split.open-file-cost | 4194304 (4 MB) | The estimated cost to open a file, used as a minimum weight when combining splits. |
+| read.parquet.vectorization.enabled| false | Enables parquet vectorization read |
+| read.parquet.vectorization.batch-size| 5000 | The batch size of parquet vectorization read |
+| read.orc.vectorization.enabled | false | Enables orc vectorization read |
+| read.orc.vectorization.batch-size | 5000 | The batch size of orc vectorization read |
### Write properties
| Property | Default | Description |
| ---------------------------------- | ------------------ | -------------------------------------------------- |
| write.format.default | parquet | Default file format for the table; parquet, avro, or orc |
+| write.delete.format.default | | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), gzip, snappy, uncompressed |
+| write.avro.compression-level | null | Avro compression level |
| write.location-provider.impl | null | Optional custom implemention for LocationProvider |
| write.metadata.compression-codec | none | Metadata compression codec; none or gzip |
| write.metadata.metrics.default | truncate(16) | Default metrics mode for all columns in the table; none, counts, truncate(length), or full |
| write.metadata.metrics.column.col1 | (not set) | Metrics mode for column 'col1' to allow per-column tuning; none, counts, truncate(length), or full |
| write.target-file-size-bytes | 536870912 (512 MB) | Controls the size of files generated to target about this many bytes |
+| write.delete.target-file-size-bytes| 67108864 (64 MB) | Controls the size of delete files generated to target about this many bytes |
| write.distribution-mode | none | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder |
+| write.delete.distribution-mode | hash | Defines distribution of write delete data |
| write.wap.enabled | false | Enables write-audit-publish writes |
| write.summary.partition-limit | 0 | Includes partition-level summary stats in snapshot summaries if the changed partition count is less than this limit |
| write.metadata.delete-after-commit.enabled | false | Controls whether to delete the oldest version metadata files after commit |
| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
| write.spark.fanout.enabled | false | Enables Partitioned-Fanout-Writer writes in Spark |
+| write.object-storage.enabled | false | Enables object storage write like s3 |
+| write.data.path | a "data" folder underneath the root path of the table | Defines the path of data files |
+| write.metadata.path | a "metadata" folder underneath the root path of the table | Defines the path of metadata files |
Review comment:
These are still subject to `LocationProvider` decisions.
I wouldn't say "defines". This is where the default location provider will place metadata files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#discussion_r808514676
##########
File path: docs/versioned/tables/configuration.md
##########
@@ -34,29 +34,50 @@ Iceberg tables support table properties to configure table behavior, like the de
| read.split.metadata-target-size | 33554432 (32 MB) | Target size when combining metadata input splits |
| read.split.planning-lookback | 10 | Number of bins to consider when combining input splits |
| read.split.open-file-cost | 4194304 (4 MB) | The estimated cost to open a file, used as a minimum weight when combining splits. |
+| read.parquet.vectorization.enabled| false | Enables parquet vectorization read |
+| read.parquet.vectorization.batch-size| 5000 | The batch size of parquet vectorization read |
+| read.orc.vectorization.enabled | false | Enables orc vectorization read |
+| read.orc.vectorization.batch-size | 5000 | The batch size of orc vectorization read |
### Write properties
| Property | Default | Description |
| ---------------------------------- | ------------------ | -------------------------------------------------- |
| write.format.default | parquet | Default file format for the table; parquet, avro, or orc |
+| write.delete.format.default | | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), gzip, snappy, uncompressed |
+| write.avro.compression-level | null | Avro compression level |
| write.location-provider.impl | null | Optional custom implemention for LocationProvider |
| write.metadata.compression-codec | none | Metadata compression codec; none or gzip |
| write.metadata.metrics.default | truncate(16) | Default metrics mode for all columns in the table; none, counts, truncate(length), or full |
| write.metadata.metrics.column.col1 | (not set) | Metrics mode for column 'col1' to allow per-column tuning; none, counts, truncate(length), or full |
| write.target-file-size-bytes | 536870912 (512 MB) | Controls the size of files generated to target about this many bytes |
+| write.delete.target-file-size-bytes| 67108864 (64 MB) | Controls the size of delete files generated to target about this many bytes |
| write.distribution-mode | none | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder |
+| write.delete.distribution-mode | hash | Defines distribution of write delete data |
| write.wap.enabled | false | Enables write-audit-publish writes |
| write.summary.partition-limit | 0 | Includes partition-level summary stats in snapshot summaries if the changed partition count is less than this limit |
| write.metadata.delete-after-commit.enabled | false | Controls whether to delete the oldest version metadata files after commit |
| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
| write.spark.fanout.enabled | false | Enables Partitioned-Fanout-Writer writes in Spark |
+| write.object-storage.enabled | false | Enables object storage write like s3 |
Review comment:
This is incorrect. S3 is enabled by using `s3://` URIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4119: Docs: modify the configuration.md
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #4119:
URL: https://github.com/apache/iceberg/pull/4119#issuecomment-1046287488
Thanks, @Zhangg7723!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org