You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2024/08/07 14:57:58 UTC

[PR] Allow setting `write.parquet.row-group-limit` [iceberg-python]

Fokko opened a new pull request, #1016:
URL: https://github.com/apache/iceberg-python/pull/1016

   And update the docs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Allow setting `write.parquet.row-group-limit` [iceberg-python]

Posted by "zhongyujiang (via GitHub)" <gi...@apache.org>.
zhongyujiang commented on code in PR #1016:
URL: https://github.com/apache/iceberg-python/pull/1016#discussion_r1710640916


##########
mkdocs/docs/configuration.md:
##########
@@ -32,6 +32,7 @@ Iceberg tables support table properties to configure table behavior.
 | -------------------------------------- | --------------------------------- | ------- | ------------------------------------------------------------------------------------------- |
 | `write.parquet.compression-codec`      | `{uncompressed,zstd,gzip,snappy}` | zstd    | Sets the Parquet compression coddec.                                                        |
 | `write.parquet.compression-level`      | Integer                           | null    | Parquet compression level for the codec. If not set, it is up to PyIceberg                  |
+| `write.parquet.row-group-limit`        | Number of rows                    | 1048576 | The upper bound of the number of entries within a single row group                          |

Review Comment:
   @Fokko @sungwy Thanks, I believe this has resolved my issue #1012 as well. 
   
   However, I would like to remind you that this option already exists in the doc, right after `write.parquet.dict-size-bytes`, the UI doesn't allow me to leave a comment there, so please expand the collapsed area to see it.
   
   Additionally, I'm kind of curious as to why the default value used this time is significantly larger than the previous one?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Allow setting `write.parquet.row-group-limit` [iceberg-python]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on PR #1016:
URL: https://github.com/apache/iceberg-python/pull/1016#issuecomment-2274320281

   LGTM @Fokko - merging in the change from main to resolve the conflict on the doc


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Allow setting `write.parquet.row-group-limit` [iceberg-python]

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #1016:
URL: https://github.com/apache/iceberg-python/pull/1016#issuecomment-2275222689

   Also threw in a test here 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Allow setting `write.parquet.row-group-limit` [iceberg-python]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy merged PR #1016:
URL: https://github.com/apache/iceberg-python/pull/1016


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Allow setting `write.parquet.row-group-limit` [iceberg-python]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on code in PR #1016:
URL: https://github.com/apache/iceberg-python/pull/1016#discussion_r1711386006


##########
mkdocs/docs/configuration.md:
##########
@@ -32,6 +32,7 @@ Iceberg tables support table properties to configure table behavior.
 | -------------------------------------- | --------------------------------- | ------- | ------------------------------------------------------------------------------------------- |
 | `write.parquet.compression-codec`      | `{uncompressed,zstd,gzip,snappy}` | zstd    | Sets the Parquet compression coddec.                                                        |
 | `write.parquet.compression-level`      | Integer                           | null    | Parquet compression level for the codec. If not set, it is up to PyIceberg                  |
+| `write.parquet.row-group-limit`        | Number of rows                    | 1048576 | The upper bound of the number of entries within a single row group                          |

Review Comment:
   Thank you for flagging this @zhongyujiang - I'll get the second one below with the older default value removed.
   
   To my understanding the new value is the correct default value that matches the default in the PyArrow ParquetWriter: https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetWriter.html



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org