You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dramaticlly (via GitHub)" <gi...@apache.org> on 2024/03/05 01:09:06 UTC

[PR] Doc: Clarify table properties on metadata metrics inferred column defaults [iceberg]

dramaticlly opened a new pull request, #9865:
URL: https://github.com/apache/iceberg/pull/9865

   Based on comment from 
   https://github.com/apache/iceberg/pull/5215/files#r915475413
   and https://github.com/apache/iceberg/pull/5215/files#r915276052
   
   100 as default limit is counted only toward top level columns, which means stats are also collected towards nested children of top level columns, but not counted in 100.
   
   Example:
   ```
   root
    |-- foo1: struct (nullable = true)
    |    |-- bar1: integer (nullable = false)
    |    |-- baz1: string (nullable = true)
    |-- foo2: struct (nullable = true)
    |    |-- bar2: integer (nullable = false)
    |    |-- baz2: string (nullable = true)
   ...
    |-- foo101: struct (nullable = true)
    |    |-- bar101: integer (nullable = true)
    |    |-- baz101: string (nullable = true)
   ```
   
   - stats for `foo1.bar1`, `foo1.baz1`, `foo2.bar2`, `foo2.baz2` will all be collected use default (truncate(16))
   - stats for `foo101.bar101`, `foo101.baz101` will be none
   
   CC @stevenzwu 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Doc: Clarify table properties on metadata metrics inferred column defaults [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #9865:
URL: https://github.com/apache/iceberg/pull/9865#issuecomment-1977787041

   thanks @dramaticlly for clarifying the doc. I have always thought the 100 limit is applicable to nested fields. didn't know it is limited to the top level.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Doc: Clarify table properties on metadata metrics inferred column defaults [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #9865:
URL: https://github.com/apache/iceberg/pull/9865#issuecomment-1997632293

   Thank you @nastra and @stevenzwu for the review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Doc: Clarify table properties on metadata metrics inferred column defaults [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9865:
URL: https://github.com/apache/iceberg/pull/9865#discussion_r1524592034


##########
docs/docs/configuration.md:
##########
@@ -61,7 +61,7 @@ Iceberg tables support table properties to configure table behavior, like the de
 | write.orc.bloom.filter.fpp                           | 0.05                        | False positive probability for Bloom filter (must > 0.0 and < 1.0)                                                                                                                                |
 | write.location-provider.impl                         | null                        | Optional custom implementation for LocationProvider                                                                                                                                               |
 | write.metadata.compression-codec                     | none                        | Metadata compression codec; none or gzip                                                                                                                                                          |
-| write.metadata.metrics.max-inferred-column-defaults  | 100                         | Defines the maximum number of columns for which metrics are collected                                                                                                                             |
+| write.metadata.metrics.max-inferred-column-defaults  | 100                         | Defines the maximum number of top level columns for which metrics are collected. Number of stored metrics can be higher than this limit for a table with nested fields                                         |

Review Comment:
   ```suggestion
   | write.metadata.metrics.max-inferred-column-defaults  | 100                         | Defines the maximum number of top level columns for which metrics are collected. Number of stored metrics can be higher than this limit for a table with nested fields                            |
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Doc: Clarify table properties on metadata metrics inferred column defaults [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #9865:
URL: https://github.com/apache/iceberg/pull/9865#issuecomment-1995068340

   @nastra @amogh-jahagirdar can you help review this as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Doc: Clarify table properties on metadata metrics inferred column defaults [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9865:
URL: https://github.com/apache/iceberg/pull/9865#discussion_r1524589743


##########
docs/docs/configuration.md:
##########
@@ -61,7 +61,7 @@ Iceberg tables support table properties to configure table behavior, like the de
 | write.orc.bloom.filter.fpp                           | 0.05                        | False positive probability for Bloom filter (must > 0.0 and < 1.0)                                                                                                                                |
 | write.location-provider.impl                         | null                        | Optional custom implementation for LocationProvider                                                                                                                                               |
 | write.metadata.compression-codec                     | none                        | Metadata compression codec; none or gzip                                                                                                                                                          |
-| write.metadata.metrics.max-inferred-column-defaults  | 100                         | Defines the maximum number of columns for which metrics are collected                                                                                                                             |
+| write.metadata.metrics.max-inferred-column-defaults  | 100                         | Defines the maximum number of top level columns for which metrics are collected. Number of stored metrics can be higher than this limit for table with nested schema                                         |

Review Comment:
   ```suggestion
   | write.metadata.metrics.max-inferred-column-defaults  | 100                         | Defines the maximum number of top level columns for which metrics are collected. Number of stored metrics can be higher than this limit for a table with nested fields                                         |
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Doc: Clarify table properties on metadata metrics inferred column defaults [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #9865:
URL: https://github.com/apache/iceberg/pull/9865


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org