You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "chenwyi2 (via GitHub)" <gi...@apache.org> on 2024/07/16 09:36:02 UTC

[I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

chenwyi2 opened a new issue, #10707:
URL: https://github.com/apache/iceberg/issues/10707

   ### Query engine
   
   spark 3.5
   iceberg 1.4.2
   
   ### Question
   
   when i set write.metadata.metrics.max-inferred-column-defaults = 10, it worked well when i write new data into table, but when i run rewrite_data_file procedure, this property doesn't work,


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "chenwyi2 (via GitHub)" <gi...@apache.org>.
chenwyi2 commented on issue #10707:
URL: https://github.com/apache/iceberg/issues/10707#issuecomment-2232166202

   normally, i set  write.metadata.metrics.max-inferred-column-defaults = 10, the metrics in manifest file such as lower_bounds, upper_bounds only has 10 keys, but when i run rewrite_data_files, the metics number has no change


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "nk1506 (via GitHub)" <gi...@apache.org>.
nk1506 commented on issue #10707:
URL: https://github.com/apache/iceberg/issues/10707#issuecomment-2230885093

   Can you share the examples? it seems it's working as expected. 
   ```
   spark-sql (default)> CALL hadoop.system.rewrite_data_files(table => 't2', options => map('min-input-files','2'));
   4	2	2480	0
   Time taken: 0.406 seconds, Fetched 1 row(s)
   spark-sql (default)> select * from hadoop.t2.files;
   0	/t2/data/c1=1/00000-12-62b70cab-1d08-407b-86b8-292add4fa0d9-0-00001.parquet	PARQUET	0	{"c1":1}	2	721	{1:76,2:73}	{1:2}	{1:0}	{}	{1:}	{1:}	NULL	[4]	NULL	0	{"c1":{"column_size":76,"value_count":2,"null_value_count":0,"nan_value_count":null,"lower_bound":1,"upper_bound":1},"c2":{"column_size":73,"value_count":null,"null_value_count":null,"nan_value_count":null,"lower_bound":null,"upper_bound":null}}
   0	/t2/data/c1=2/00000-11-1937d205-9260-4555-a5d8-cb148a1562e0-0-00001.parquet	PARQUET	0	{"c1":2}	2	721	{1:76,2:73}	{1:2}	{1:0}	{}	{1:}	{1:}	NULL	[4]	NULL	0	{"c1":{"column_size":76,"value_count":2,"null_value_count":0,"nan_value_count":null,"lower_bound":2,"upper_bound":2},"c2":{"column_size":73,"value_count":null,"null_value_count":null,"nan_value_count":null,"lower_bound":null,"upper_bound":null}}
   Time taken: 0.069 seconds, Fetched 2 row(s)
   spark-sql (default)>
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "chenwyi2 (via GitHub)" <gi...@apache.org>.
chenwyi2 closed issue #10707: write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file?
URL: https://github.com/apache/iceberg/issues/10707


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "chenwyi2 (via GitHub)" <gi...@apache.org>.
chenwyi2 commented on issue #10707:
URL: https://github.com/apache/iceberg/issues/10707#issuecomment-2241984594

   oh i see, thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "chenwyi2 (via GitHub)" <gi...@apache.org>.
chenwyi2 commented on issue #10707:
URL: https://github.com/apache/iceberg/issues/10707#issuecomment-2232539561

   > {"status":1,"snapshot_id":{"long":5538718232046950695},"data_file":{"file_path":"xxxxx/xxxx00-7-7ed40c8d-6d72-495e-925e-256e45434b5b-xxxx01.parquet","file_format":"PARQUET","partition":{"dt":{"string":"2024-06-08"},"hour":{"string":"02"},"timestamp":{"long":17177862xxxx00},"bucket_bucket":{"int":4}},"record_count":xxx,"file_size_in_bytes":xxx,"block_size_in_bytes":xxx,"column_sizes":{"array":[{"key":1,"value":xxx},{"key":2,"value":xxx},{"key":3,"value":xxx},{"key":4,"value":xxx},{"key":5,"value":139525},{"key":6,"value":197154},{"key":7,"value":460660},{"key":8,"value":1057662},{"key":9,"value":8586936},{"key":10,"value":260495},{"key":11,"value":146196},{"key":12,"value":116454},{"key":13,"value":229965},{"key":14,"value":186363},{"key":15,"value":264108},{"key":16,"value":323974},{"key":17,"value":518138},{"key":18,"value":4088},{"key":19,"value":235623},{"key":20,"value":3798},{"key":21,"value":8392},{"key":22,"value":399474},{"key":23,"value":112300},{"key":24,"value":2899},
 {"key":25,"value":155199},{"key":26,"value":8748},{"key":27,"value":467972},{"key":28,"value":364833},{"key":29,"value":9770},{"key":30,"value":14327},{"key":31,"value":86767},{"key":32,"value":122381},{"key":33,"value":541669},{"key":34,"value":58329},{"key":35,"value":3925},{"key":36,"value":159908},{"key":37,"value":8567757},{"key":38,"value":426402},{"key":39,"value":706207},{"key":40,"value":1838658},{"key":41,"value":267520},{"key":42,"value":2340058},{"key":43,"value":1217},{"key":44,"value":1079238},{"key":45,"value":24149},{"key":46,"value":450576},{"key":47,"value":219157},{"key":48,"value":57122},{"key":49,"value":456931},{"key":50,"value":82592},{"key":51,"value":1217},{"key":52,"value":47612},{"key":53,"value":48266},{"key":54,"value":6426},{"key":55,"value":60071},{"key":56,"value":60394},{"key":57,"value":17145},{"key":58,"value":1217},{"key":59,"value":123414},{"key":60,"value":90264},{"key":61,"value":52875},{"key":62,"value":62797},{"key":63,"value":6694134},{"key"
 :64,"value":17049},{"key":65,"value":116783},{"key":66,"value":134637},{"key":67,"value":176748},{"key":68,"value":138273},{"key":69,"value":3012},{"key":70,"value":140782},{"key":71,"value":128007},{"key":72,"value":23168},{"key":73,"value":25340},{"key":74,"value":1305},{"key":75,"value":416956},{"key":76,"value":1565},{"key":77,"value":32529},{"key":78,"value":1217},{"key":79,"value":1317},{"key":80,"value":1864},{"key":81,"value":1863},{"key":82,"value":1360},{"key":83,"value":1387},{"key":84,"value":6490},{"key":85,"value":1863},{"key":86,"value":1863},{"key":87,"value":1889},{"key":88,"value":1945},{"key":89,"value":1604},{"key":90,"value":1604},{"key":91,"value":1304},{"key":92,"value":4720},{"key":93,"value":1217},{"key":94,"value":2905601},{"key":95,"value":10826},{"key":96,"value":93052},{"key":97,"value":1304},{"key":98,"value":1628},{"key":99,"value":1994},{"key":100,"value":80275},{"key":105,"value":18524787},{"key":106,"value":55202},{"key":108,"value":180561},{"key":1
 09,"value":1315},{"key":110,"value":1307},{"key":111,"value":1309},{"key":112,"value":3798},{"key":113,"value":2599529},{"key":114,"value":32606537},{"key":115,"value":368977},{"key":116,"value":1073790},{"key":117,"value":4536},{"key":118,"value":4676},{"key":119,"value":54506},{"key":120,"value":92709},{"key":123,"value":133171},{"key":124,"value":2921932},{"key":125,"value":749791}]},"value_counts":{"array":[{"key":1,"value":443627},{"key":2,"value":443627},{"key":3,"value":443627},{"key":4,"value":443627},{"key":5,"value":443627},{"key":6,"value":443627},{"key":7,"value":443627},{"key":8,"value":443627},{"key":9,"value":443627},{"key":10,"value":443627},{"key":11,"value":443627},{"key":12,"value":443627},{"key":13,"value":443627},{"key":14,"value":443627},{"key":15,"value":443627},{"key":16,"value":443627},{"key":17,"value":443627},{"key":18,"value":443627},{"key":19,"value":443627},{"key":20,"value":443627},{"key":21,"value":443627},{"key":22,"value":443627},{"key":23,"value":4
 43627},{"key":24,"value":443627},{"key":25,"value":443627},{"key":26,"value":443627},{"key":27,"value":443627},{"key":28,"value":443627},{"key":29,"value":443627},{"key":30,"value":443627},{"key":31,"value":443627},{"key":32,"value":443627},{"key":33,"value":443627},{"key":34,"value":443627},{"key":35,"value":443627},{"key":36,"value":443627},{"key":37,"value":443627},{"key":38,"value":443627},{"key":39,"value":443627},{"key":40,"value":443627},{"key":41,"value":443627},{"key":42,"value":443627},{"key":43,"value":443627},{"key":44,"value":443627},{"key":45,"value":443627},{"key":46,"value":443627},{"key":47,"value":443627},{"key":48,"value":443627},{"key":49,"value":443627},{"key":50,"value":443627},{"key":51,"value":443627},{"key":52,"value":443627},{"key":53,"value":443627},{"key":54,"value":443627},{"key":55,"value":443627},{"key":56,"value":443627},{"key":57,"value":443627},{"key":58,"value":443627},{"key":59,"value":443627},{"key":60,"value":443627},{"key":61,"value":443627},{"
 key":62,"value":443627},{"key":63,"value":443627},{"key":64,"value":443627},{"key":65,"value":443627},{"key":66,"value":443627},{"key":67,"value":443627},{"key":68,"value":443627},{"key":69,"value":443627},{"key":70,"value":443627},{"key":71,"value":443627},{"key":72,"value":443627},{"key":73,"value":443627},{"key":74,"value":443627},{"key":75,"value":443627},{"key":76,"value":443627},{"key":77,"value":443627},{"key":78,"value":443627},{"key":79,"value":443627},{"key":80,"value":443627},{"key":81,"value":443627},{"key":82,"value":443627},{"key":83,"value":443627},{"key":84,"value":443627},{"key":85,"value":443627},{"key":86,"value":443627},{"key":87,"value":443627},{"key":88,"value":443627},{"key":89,"value":443627},{"key":90,"value":443627},{"key":91,"value":443627},{"key":92,"value":443627},{"key":93,"value":443627},{"key":94,"value":443627},{"key":95,"value":443627},{"key":96,"value":443627},{"key":97,"value":443627},{"key":98,"value":443627},{"key":99,"value":443627},{"key":100,
 "value":443627},{"key":105,"value":443627},{"key":106,"value":443627},{"key":108,"value":443627},{"key":109,"value":443627},{"key":110,"value":443627},{"key":111,"value":443627},{"key":112,"value":443627},{"key":113,"value":16984783},{"key":114,"value":16984783},{"key":115,"value":933535},{"key":116,"value":933535},{"key":117,"value":443627},{"key":118,"value":443627},{"key":119,"value":459918},{"key":120,"value":459918},{"key":123,"value":517904},{"key":124,"value":517904},{"key":125,"value":443627}]},"null_value_counts":{"array":[{"key":1,"value":0},{"key":2,"value":0},{"key":3,"value":43119},{"key":4,"value":4335},{"key":5,"value":0},{"key":6,"value":142765},{"key":7,"value":212855},{"key":8,"value":176377},{"key":9,"value":10385},{"key":10,"value":0},{"key":11,"value":0},{"key":12,"value":0},{"key":13,"value":0},{"key":14,"value":0},{"key":15,"value":334261},{"key":16,"value":121},{"key":17,"value":121},{"key":18,"value":0},{"key":19,"value":0},{"key":20,"value":0},{"key":21,"va
 lue":0},{"key":22,"value":117047},{"key":23,"value":383157},{"key":24,"value":443172},{"key":25,"value":20},{"key":26,"value":121},{"key":27,"value":2480},{"key":28,"value":1},{"key":29,"value":0},{"key":30,"value":0},{"key":31,"value":0},{"key":32,"value":0},{"key":33,"value":0},{"key":34,"value":0},{"key":35,"value":443341},{"key":36,"value":0},{"key":37,"value":6338},{"key":38,"value":0},{"key":39,"value":407682},{"key":40,"value":360859},{"key":41,"value":428598},{"key":42,"value":220719},{"key":43,"value":443627},{"key":44,"value":318327},{"key":45,"value":0},{"key":46,"value":0},{"key":47,"value":60470},{"key":48,"value":0},{"key":49,"value":0},{"key":50,"value":357133},{"key":51,"value":443627},{"key":52,"value":47},{"key":53,"value":383157},{"key":54,"value":441894},{"key":55,"value":432133},{"key":56,"value":432133},{"key":57,"value":0},{"key":58,"value":443627},{"key":59,"value":0},{"key":60,"value":0},{"key":61,"value":408949},{"key":62,"value":38506},{"key":63,"value":26
 6275},{"key":64,"value":435548},{"key":65,"value":133941},{"key":66,"value":133941},{"key":67,"value":129512},{"key":68,"value":133949},{"key":69,"value":0},{"key":70,"value":260241},{"key":71,"value":297708},{"key":72,"value":0},{"key":73,"value":0},{"key":74,"value":0},{"key":75,"value":31867},{"key":76,"value":0},{"key":77,"value":429441},{"key":78,"value":443627},{"key":79,"value":443619},{"key":80,"value":443477},{"key":81,"value":443477},{"key":82,"value":0},{"key":83,"value":0},{"key":84,"value":439},{"key":85,"value":443477},{"key":86,"value":443477},{"key":87,"value":443477},{"key":88,"value":0},{"key":89,"value":0},{"key":90,"value":0},{"key":91,"value":0},{"key":92,"value":443158},{"key":93,"value":443627},{"key":94,"value":300346},{"key":95,"value":0},{"key":96,"value":0},{"key":97,"value":0},{"key":98,"value":0},{"key":99,"value":0},{"key":100,"value":0},{"key":105,"value":52},{"key":106,"value":342663},{"key":108,"value":383157},{"key":109,"value":0},{"key":110,"value"
 :0},{"key":111,"value":0},{"key":112,"value":0},{"key":113,"value":0},{"key":114,"value":0},{"key":115,"value":46288},{"key":116,"value":46288},{"key":117,"value":442685},{"key":118,"value":442685},{"key":119,"value":427080},{"key":120,"value":427080},{"key":123,"value":369350},{"key":124,"value":369350},{"key":125,"value":32127}]},"nan_value_counts":{"array":[{"key":66,"value":0},{"key":98,"value":0},{"key":99,"value":0},{"key":68,"value":0},{"key":23,"value":0},{"key":55,"value":0},{"key":56,"value":0}]},"lower_bounds":{"array":[{"key":1,"value":"xxx"},{"key":2,"value":"xxxx"},{"key":3,"value":"0"},{"key":4,"value":"xxx"},{"key":5,"value":"xxxx"},{"key":6,"value":"xxxxx"},{"key":7,"value":"xxxxx"},{"key":8,"value":"4ª]\uxxxx5\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":9,"value":"xxxx01d075833800beda83896cd060971"},{"key":10,"value":"\uxxxxF\u0010¥Ôè\uxxxx0\uxxxx0\uxxxx0"},{"key":11,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":12,"value":"1"},{"key":13,"value":"qc_1xxxx01_1xxxx02"}
 ,{"key":14,"value":"G\u0010¥Ôè\uxxxx0\uxxxx0\uxxxx0"},{"key":15,"value":"ç:\uxxxx5\uxxxx6\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":16,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":17,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":18,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":19,"value":"1903-10-08 23:17:15"},{"key":20,"value":"1717786206"},{"key":21,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":22,"value":"-1"},{"key":23,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":24,"value":"undefined"},{"key":25,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":26,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":27,"value":"1.0.0"},{"key":28,"value":"1.0.4-ares6-pure"},{"key":29,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":30,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":31,"value":"\uxxxx0\uxxxx0\
 uxxxx0\uxxxx0"},{"key":32,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":33,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":34,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":35,"value":"0:1;76:1;538:6;502:2;528:10;"},{"key":36,"value":"1"},{"key":37,"value":"xxxx00f55625446dddaa057dcc7d2cc3a"},{"key":38,"value":"­Ö\u0012d\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":39,"value":"xxxxxxxx00%2Dxxxx0%2Dxxxx0%2Dxxxx0%2Dxxxxxxxxxxxxxxxx"},{"key":40,"value":"xxxx0b0d5a6c3785c96e260d59c328512ffda51a1"},{"key":41,"value":"xxxx1f12c5209e46b253ad53b72213aaa"},{"key":42,"value":"xxxxxxxxxxxxxxxxxxxx0"},{"key":44,"value":"00"},{"key":45,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":46,"value":"ÿÿÿÿÿÿÿÿ"},{"key":47,"value":"1:1001:"},{"key":48,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":49,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":50,"value":"b01"},{"key":52,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":53,"value":"ÿÿÿÿÿÿÿÿ"},{"key":54,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"
 key":55,"value":"Æú\uxxxx6&7Êò¿"},{"key":56,"value":"‚\uxxxx3Zº‚qBÀ"},{"key":57,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":59,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":60,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":61,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":62,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":63,"value":"003e3ab26e82482288908fd7fbc0ac61"},{"key":64,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":65,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":66,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":67,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":68,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":69,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":70,"value":"0"},{"key":71,"value":"-1465858398"},{"key":72,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":73,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":74,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":75,"value":"-g"},{"key":76,"value":"\uxxxx0\u
 xxxx0\uxxxx0\uxxxx0"},{"key":77,"value":"1"},{"key":79,"value":"11"},{"key":80,"value":"CUPID_NFID"},{"key":81,"value":"CUPID_NFDS"},{"key":82,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":83,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":84,"value":"CUPID_CLA"},{"key":85,"value":"CUPID_DCS"},{"key":86,"value":"CUPID_DCI"},{"key":87,"value":"1"},{"key":88,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":89,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":90,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":91,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":92,"value":"1088238243247739"},{"key":94,"value":"RUSPS009898b7244f4da62c68342517fe26ca"},{"key":95,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":96,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":97,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":98,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":99,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":100,"value
 ":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":105,"value":"xxxx"},{"key":106,"value":"\uxxxx2\uxxxx0\uxxxx0\uxxxx0"},{"key":108,"value":"{\"payment_type\":\"1\",\"deliver_type\":\"100\",\"origin_order_item_id\":\"5xxxx001327650\",\"is_ssp_secret_release\":\"0\",\"price\":\"0\",\"billing_type\":\"2\",\"is_regional_brand\":\"0\",\"weight\":\"88\",\"order_id\":\"5xxxxxxxx610811\",\"advertiser_id\":\"71xxxxxxxx2031\"}"},{"key":109,"value":"2024-06-08"},{"key":110,"value":"02"},{"key":111,"value":"ÀG\tô\uxxxx1\uxxxx0\uxxxx0"},{"key":112,"value":"1717786206"},{"key":125,"value":"%3DMV100-YTS-8188"}]},"upper_bounds":{"array":[{"key":1,"value":"fffff7247370043d7885cf392fe93867"},{"key":2,"value":"99.92.208.73"},{"key":3,"value":"CUPID_PPID"},{"key":4,"value":"ztm2zzxlqf5fx2qvsejie6e4nlkwkmfe"},{"key":5,"value":"8\u0018\uxxxx0\uxxxx0"},{"key":6,"value":"q\u0011\uxxxx1\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":7,"value":"©¡\rËsù\u001F\uxxxx0"},{"key":8,"value":"|L\nÒsù\u001F\uxxxx0"},{
 "key":9,"value":"ffffef296651b908d95219cf19989fbb"},{"key":10,"value":"€‘ë\u0015\r*\uxxxx6\uxxxx0"},{"key":11,"value":"\u0010\uxxxx0\uxxxx0\uxxxx0"},{"key":12,"value":"Z"},{"key":13,"value":"qc_105394_101620"},{"key":14,"value":"чW'Œ\uxxxx4\uxxxx0\uxxxx0"},{"key":15,"value":"3\u001DY'Œ\uxxxx4\uxxxx0\uxxxx0"},{"key":16,"value":"ã…\uxxxx1\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":17,"value":"8ó};\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":18,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":19,"value":"2034-11-13 20:39:59"},{"key":20,"value":"1717786496"},{"key":21,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":22,"value":"zzfy@xt001"},{"key":23,"value":"\uxxxxE\uxxxxEYC"},{"key":24,"value":"undefined"},{"key":25,"value":"G\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":26,"value":"æ\uxxxx3\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":27,"value":"v2024060390(MiVideo-UN)"},{"key":28,"value":"ares-baidu-1.21.0
 "},{"key":29,"value":" \uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":30,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":31,"value":"Ï\uxxxx7\uxxxx0\uxxxx0"},{"key":32,"value":"D\u0015\uxxxx0\uxxxx0"},{"key":33,"value":"p\uxxxx0\uxxxx0"},{"key":34,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":35,"value":"77:1;16:7;15:5;3:19;36:18;10:21;38:3;0:112;543:6;75:7;525:4;76:31;538:15;571:1;504:11;558:2;1146:26;502:180;"},{"key":36,"value":"2;3;4;5;6;7;8;9;A;B"},{"key":37,"value":"ffffc834d988c4c40ceb86f724ce06b8"},{"key":38,"value":"€Wcf\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":39,"value":"FFFE04B4-E402-49E4-BECC-F6F726DA3ADE"},{"key":40,"value":"xxx"},{"key":41,"value":"fff684bf3348d3c236d06973f2aa6499"},{"key":42,"value":"fffffa24cf026cca"},{"key":44,"value":"fe:ee:81:d0:88:c9"},{"key":45,"value":"N\uxxxx2\uxxxx0\uxxxx0"},{"key":46,"value":"8 ²Ôè\uxxxx0\uxxxx0\uxxxx0"},{"key":47,"value":"Z:2002:2046"},{"key":48,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":49,"value":"f)ã
 \uxxxx0"},{"key":50,"value":"x00"},{"key":52,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":53,"value":"ÿÿÿÿÿÿÿÿ"},{"key":54,"value":"\uxxxx2\uxxxx0\uxxxx0\uxxxx0"},{"key":55,"value":"nmáy)Ýe@"},{"key":56,"value":"€Ÿqá@º[@"},{"key":57,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":59,"value":"\uxxxx0 \uxxxx0\uxxxx0"},{"key":60,"value":"@\uxxxx0\uxxxx0\uxxxx0"},{"key":61,"value":"´\uxxxx2\uxxxx0\uxxxx0"},{"key":62,"value":"\uxxxx1\b\uxxxx0\uxxxx0"},{"key":63,"value":"fff67654884f485b984524d41cf28eaf"},{"key":64,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":65,"value":"\uxxxx2\uxxxx0\uxxxx0\uxxxx0"},{"key":66,"value":"\uxxxx0\uxxxx0\uxxxx0`¸\u001Eí?"},{"key":67,"value":"^\uxxxx2\uxxxx0\uxxxx0"},{"key":68,"value":"\uxxxx0\uxxxx0\uxxxx0`¸\u001Eí?"},{"key":69,"value":"\b\uxxxx0\uxxxx0\uxxxx0"},{"key":70,"value":"99999"},{"key":71,"value":"99999"},{"key":72,"value":"e\uxxxx2\uxxxx0\uxxxx0"},{"key":73,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":74,"value":"\uxxxx1\ux
 xxx0\uxxxx0\uxxxx0"},{"key":75,"value":"首趣@S1"},{"key":76,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":77,"value":"99"},{"key":79,"value":"34056"},{"key":80,"value":"CUPID_NFID"},{"key":81,"value":"CUPID_NFDS"},{"key":82,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":83,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":84,"value":"subrange"},{"key":85,"value":"CUPID_DCS"},{"key":86,"value":"CUPID_DCI"},{"key":87,"value":"CUPID_FC"},{"key":88,"value":"\uxxxx2\uxxxx0\uxxxx0\uxxxx0"},{"key":89,"value":"!N\uxxxx0\uxxxx0"},{"key":90,"value":"!N\uxxxx0\uxxxx0"},{"key":91,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":92,"value":"8934222399602739"},{"key":94,"value":"null"},{"key":95,"value":"\uxxxx1\uxxxx0\uxxxx0\uxxxx0"},{"key":96,"value":"\n\uxxxx0\uxxxx0\uxxxx0"},{"key":97,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0"},{"key":98,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0à?"},{"key":99,"value":"\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0\uxxxx0ð?"},{"key
 ":100,"value":"\uxxxx5\uxxxx0\uxxxx0\uxxxx0"},{"key":105,"value":"xxxx"},{"key":106,"value":"\uxxxx2\uxxxx0\uxxxx0\uxxxx0"},{"key":108,"value":"{\"payment_type\":\"2\",\"deliver_type\":\"22\",\"origin_order_item_id\":\"5xxxx001977893\",\"is_ssp_secret_release\":\"0\",\"price\":\"1xxxx\",\"billing_type\":\"2\",\"is_regional_brand\":\"0\",\"weight\":\"71\",\"order_id\":\"5xxxxxxxx646961\",\"advertiser_id\":\"171xxxx0026086\"}"},{"key":109,"value":"2024-06-08"},{"key":110,"value":"02"},{"key":111,"value":"ÀG\tô\uxxxx1\uxxxx0\uxxxx0"},{"key":112,"value":"xxxx"},{"key":125,"value":"zy-6s"}]},"key_metadata":null,"split_offsets":{"array":[4]},"sort_order_id":{"int":0}}}


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "nk1506 (via GitHub)" <gi...@apache.org>.
nk1506 commented on issue #10707:
URL: https://github.com/apache/iceberg/issues/10707#issuecomment-2232443565

   @chenwyi2 , Can you please share the manifest file content after `rewrite_data_files` ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "chenwyi2 (via GitHub)" <gi...@apache.org>.
chenwyi2 commented on issue #10707:
URL: https://github.com/apache/iceberg/issues/10707#issuecomment-2235150133

   `  "write.metadata.metrics.max-inferred-column-defaults" : "10",
       "write.watermark.field" : "timestamp",
       "write.metadata.delete-after-commit.enabled" : "true",
       "write.distribution-mode" : "hash",
       "write.metadata.metrics.default" : "full",
       "write.summary.partition-limit" : "100"`
   
   > If you have configured metrics mode write.metadata.metrics.default then it might not be honoring write.metadata.metrics.max-inferred-column-defaults
   what's the meaning ?   in my opinion, write.metadata.metrics.default is the default metrics mode for all columns in the table; none, counts, truncate(length), or full


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "nk1506 (via GitHub)" <gi...@apache.org>.
nk1506 commented on issue #10707:
URL: https://github.com/apache/iceberg/issues/10707#issuecomment-2236218110

   Hi @chenwyi2 , As I mentioned before if `write.metadata.metrics.default` has been configured it won't honour  `write.metadata.metrics.max-inferred-column-defaults`. Since in your case `write.metadata.metrics.default` has been configured with value `full`. That's why it is not considering `write.metadata.metrics.max-inferred-column-defaults`. 
   You can refer here https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MetricsConfig.java#L125. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "chenwyi2 (via GitHub)" <gi...@apache.org>.
chenwyi2 commented on issue #10707:
URL: https://github.com/apache/iceberg/issues/10707#issuecomment-2232535366

   `
   ![image](https://github.com/user-attachments/assets/9d9ccfd2-9b41-474f-bc61-577aaa2b7a01)
   `


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "chenwyi2 (via GitHub)" <gi...@apache.org>.
chenwyi2 commented on issue #10707:
URL: https://github.com/apache/iceberg/issues/10707#issuecomment-2232537932

   >![image](https://github.com/user-attachments/assets/7b703884-6398-4668-b217-3802a128858c)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.metadata.metrics.max-inferred-column-defaults doesn't work for rewrite_data_file? [iceberg]

Posted by "nk1506 (via GitHub)" <gi...@apache.org>.
nk1506 commented on issue #10707:
URL: https://github.com/apache/iceberg/issues/10707#issuecomment-2233394203

   @chenwyi2 , Please share the table properties from metadata json. 
   something like 
   ```
   "properties" : {
       "owner" : "<>",
       "write.metadata.metrics.max-inferred-column-defaults" : "1",
       "write.metadata.metrics.default" : "truncate(2)",
       "write.parquet.compression-codec" : "zstd"
     }
   ```
   If you have configured metrics mode `write.metadata.metrics.default`  then it might not be honoring  `write.metadata.metrics.max-inferred-column-defaults`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org