You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/06/29 10:00:32 UTC
[GitHub] [iceberg] renshangtao opened a new pull request, #5158: Doc:Add parquet,orc,avro delete codec and level
renshangtao opened a new pull request, #5158:
URL: https://github.com/apache/iceberg/pull/5158
Add the parquet,orc,avro delete codec and level properties in configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #5158: Doc:Add parquet,orc,avro delete codec and level
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#issuecomment-1207452823
I think it is good to note that the default is the data value, but document the settings exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] renshangtao commented on a diff in pull request #5158: Doc:Add parquet,orc,avro delete codec and level
Posted by GitBox <gi...@apache.org>.
renshangtao commented on code in PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#discussion_r910549790
##########
docs/configuration.md:
##########
@@ -72,7 +82,7 @@ Iceberg tables support table properties to configure table behavior, like the de
| write.wap.enabled | false | Enables write-audit-publish writes |
| write.summary.partition-limit | 0 | Includes partition-level summary stats in snapshot summaries if the changed partition count is less than this limit |
| write.metadata.delete-after-commit.enabled | false | Controls whether to delete the oldest version metadata files after commit |
-| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
+| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
Review Comment:
Thank you for your reply. I have had revert the whitespace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] renshangtao commented on pull request #5158: Doc:Add parquet,orc,avro delete codec and level
Posted by GitBox <gi...@apache.org>.
renshangtao commented on PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#issuecomment-1173271704
@hililiwei Yes, you are right.The parquet and avro is the same as orc.
``
String codecAsString = config.get(DELETE_PARQUET_COMPRESSION);
``
``
CompressionCodecName codec = codecAsString != null ? toCodec(codecAsString) : dataContext.codec();
``
@rdblue What is your opinion? Do we need to expose them.
Can we modify it like this
Property | Default | Description
-- | -- | --
write.delete.orc.compression-codec | data compression codec | ORC compression codec: zstd, lz4, lzo, zlib, snappy, none
In the current document the write.delete format.default was exposed
Property | Default | Description
-- | -- | --
write.format.default | parquet | Default file format for the table; parquet, avro, or orc
write.delete.format.default | data file format | Default delete file format for the table; parquet, avro, or orc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5158: Doc:Add parquet,orc,avro delete codec and level
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#discussion_r909838861
##########
docs/configuration.md:
##########
@@ -72,7 +82,7 @@ Iceberg tables support table properties to configure table behavior, like the de
| write.wap.enabled | false | Enables write-audit-publish writes |
| write.summary.partition-limit | 0 | Includes partition-level summary stats in snapshot summaries if the changed partition count is less than this limit |
| write.metadata.delete-after-commit.enabled | false | Controls whether to delete the oldest version metadata files after commit |
-| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
+| write.metadata.previous-versions-max | 100 | The max number of previous version metadata files to keep before deleting after commit |
Review Comment:
Can you revert this whitespace-only change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] renshangtao commented on a diff in pull request #5158: Doc:Add parquet,orc,avro delete codec and level
Posted by GitBox <gi...@apache.org>.
renshangtao commented on code in PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#discussion_r910532491
##########
docs/configuration.md:
##########
@@ -52,15 +52,25 @@ Iceberg tables support table properties to configure table behavior, like the de
| write.delete.format.default | data file format | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
+| write.delete.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
+| write.delete.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
+| write.delete.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
+| write.delete.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), zstd, snappy, uncompressed |
| write.avro.compression-level | null | Avro compression level |
+| write.delete.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), zstd, snappy, uncompressed |
+| write.delete.avro.compression-level | null | Avro compression level |
| write.orc.stripe-size-bytes | 67108864 (64 MB) | Define the default ORC stripe size, in bytes |
+| write.delete.orc.stripe-size-bytes | 67108864 (64 MB) | Define the default ORC stripe size, in bytes |
| write.orc.block-size-bytes | 268435456 (256 MB) | Define the default file system block size for ORC files |
+| write.delete.orc.block-size-bytes | 268435456 (256 MB) | Define the default file system block size for ORC files |
| write.orc.compression-codec | zlib | ORC compression codec: zstd, lz4, lzo, zlib, snappy, none |
| write.orc.compression-strategy | speed | ORC compression strategy: speed, compression |
+| write.delete.orc.compression-codec | zlib | ORC compression codec: zstd, lz4, lzo, zlib, snappy, none |
+| write.delete.orc.compression-strategy | speed | ORC compression strategy: speed, compression |
Review Comment:
yes the delete is the same as write.orc.com press-codec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc:Add parquet,orc,avro delete codec and level [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#issuecomment-2309064100
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc:Add parquet,orc,avro delete codec and level [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5158: Doc:Add parquet,orc,avro delete codec and level
URL: https://github.com/apache/iceberg/pull/5158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5158: Doc:Add parquet,orc,avro delete codec and level
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#discussion_r909840565
##########
docs/configuration.md:
##########
@@ -52,15 +52,25 @@ Iceberg tables support table properties to configure table behavior, like the de
| write.delete.format.default | data file format | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
+| write.delete.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
+| write.delete.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
+| write.delete.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
+| write.delete.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), zstd, snappy, uncompressed |
| write.avro.compression-level | null | Avro compression level |
+| write.delete.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), zstd, snappy, uncompressed |
+| write.delete.avro.compression-level | null | Avro compression level |
| write.orc.stripe-size-bytes | 67108864 (64 MB) | Define the default ORC stripe size, in bytes |
+| write.delete.orc.stripe-size-bytes | 67108864 (64 MB) | Define the default ORC stripe size, in bytes |
| write.orc.block-size-bytes | 268435456 (256 MB) | Define the default file system block size for ORC files |
+| write.delete.orc.block-size-bytes | 268435456 (256 MB) | Define the default file system block size for ORC files |
| write.orc.compression-codec | zlib | ORC compression codec: zstd, lz4, lzo, zlib, snappy, none |
| write.orc.compression-strategy | speed | ORC compression strategy: speed, compression |
+| write.delete.orc.compression-codec | zlib | ORC compression codec: zstd, lz4, lzo, zlib, snappy, none |
+| write.delete.orc.compression-strategy | speed | ORC compression strategy: speed, compression |
Review Comment:
Are these all supported options?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5158: Doc:Add parquet,orc,avro delete codec and level
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#discussion_r909847636
##########
docs/configuration.md:
##########
@@ -52,15 +52,25 @@ Iceberg tables support table properties to configure table behavior, like the de
| write.delete.format.default | data file format | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
+| write.delete.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
+| write.delete.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
+| write.delete.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
+| write.delete.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), zstd, snappy, uncompressed |
| write.avro.compression-level | null | Avro compression level |
+| write.delete.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), zstd, snappy, uncompressed |
Review Comment:
Clauses within parens should be separated from the preceding words by a space: `gzip (deflate ...)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc:Add parquet,orc,avro delete codec and level [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#issuecomment-2292492463
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] renshangtao commented on a diff in pull request #5158: Doc:Add parquet,orc,avro delete codec and level
Posted by GitBox <gi...@apache.org>.
renshangtao commented on code in PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#discussion_r910550000
##########
docs/configuration.md:
##########
@@ -52,15 +52,25 @@ Iceberg tables support table properties to configure table behavior, like the de
| write.delete.format.default | data file format | Default delete file format for the table; parquet, avro, or orc |
| write.parquet.row-group-size-bytes | 134217728 (128 MB) | Parquet row group size |
| write.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
+| write.delete.parquet.page-size-bytes | 1048576 (1 MB) | Parquet page size |
| write.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
+| write.delete.parquet.dict-size-bytes | 2097152 (2 MB) | Parquet dictionary page size |
| write.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
| write.parquet.compression-level | null | Parquet compression level |
+| write.delete.parquet.compression-codec | gzip | Parquet compression codec: zstd, brotli, lz4, gzip, snappy, uncompressed |
+| write.delete.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), zstd, snappy, uncompressed |
| write.avro.compression-level | null | Avro compression level |
+| write.delete.avro.compression-codec | gzip | Avro compression codec: gzip(deflate with 9 level), zstd, snappy, uncompressed |
Review Comment:
I have modified it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on pull request #5158: Doc:Add parquet,orc,avro delete codec and level
Posted by GitBox <gi...@apache.org>.
hililiwei commented on PR #5158:
URL: https://github.com/apache/iceberg/pull/5158#issuecomment-1172825045
Although we can configure these for the DELETE separately, I do not recommend exposing them.
If we must do so, its default value is consistent with data file.
e.g.:
`long stripeSize = PropertyUtil.propertyAsLong(config, DELETE_ORC_STRIPE_SIZE_BYTES, dataContext.stripeSize());`
So, maybe it should look like this:
| Property | Default | Description |
| ---------------------------------- | ------------------ | -------------------------------------------------- |
| write.delete.orc.compression-codec | write.orc.compression-codec | ORC compression codec: zstd, lz4, lzo, zlib, snappy, none |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org