You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "manirajv06 (via GitHub)" <gi...@apache.org> on 2025/09/20 08:25:41 UTC
[PR] Collect list of columns written in file [iceberg]
manirajv06 opened a new pull request, #14126:
URL: https://github.com/apache/iceberg/pull/14126
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Collect list of columns written in file [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14126:
URL: https://github.com/apache/iceberg/pull/14126#issuecomment-3644352736
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Collect list of columns written in file [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #14126:
URL: https://github.com/apache/iceberg/pull/14126#discussion_r2373077398
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetrics.java:
##########
@@ -87,6 +92,7 @@ static Metrics metrics(
}
int fieldId = id.intValue();
+ columnsWritten.add(fieldId);
Review Comment:
Does this work with columns which are read via name mappings?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Collect list of columns written in file [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14126:
URL: https://github.com/apache/iceberg/pull/14126#issuecomment-3505454233
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Collect list of columns written in file [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14126: Collect list of columns written in file
URL: https://github.com/apache/iceberg/pull/14126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Collect list of columns written in file [iceberg]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #14126:
URL: https://github.com/apache/iceberg/pull/14126#discussion_r2373835440
##########
api/src/main/java/org/apache/iceberg/Metrics.java:
##########
@@ -179,6 +206,7 @@ Map<Integer, Type> originalTypes() {
*/
private void writeObject(ObjectOutputStream out) throws IOException {
out.writeObject(rowCount);
+ out.writeObject(columnsWritten);
Review Comment:
It seems the orders between readObject() and writeObject() mismatch.
- writeObject writes: rowCount → columnsWritten → columnSizes → ...
- readObject reads: rowCount → columnSizes → columnsWritten → ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Collect list of columns written in file [iceberg]
Posted by "manirajv06 (via GitHub)" <gi...@apache.org>.
manirajv06 commented on PR #14126:
URL: https://github.com/apache/iceberg/pull/14126#issuecomment-3517578529
recheck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Collect list of columns written in file [iceberg]
Posted by "manirajv06 (via GitHub)" <gi...@apache.org>.
manirajv06 commented on code in PR #14126:
URL: https://github.com/apache/iceberg/pull/14126#discussion_r2376502162
##########
api/src/main/java/org/apache/iceberg/Metrics.java:
##########
@@ -179,6 +206,7 @@ Map<Integer, Type> originalTypes() {
*/
private void writeObject(ObjectOutputStream out) throws IOException {
out.writeObject(rowCount);
+ out.writeObject(columnsWritten);
Review Comment:
Yes, you are right. Raised this PR as draft quickly to confirm the approach. Will correct during the real impl. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Collect list of columns written in file [iceberg]
Posted by "manirajv06 (via GitHub)" <gi...@apache.org>.
manirajv06 commented on code in PR #14126:
URL: https://github.com/apache/iceberg/pull/14126#discussion_r2376497638
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetrics.java:
##########
@@ -87,6 +92,7 @@ static Metrics metrics(
}
int fieldId = id.intValue();
+ columnsWritten.add(fieldId);
Review Comment:
My understanding is, More than one name could be mapped to the same field id. Say, Name1 & Name2 mapped to field id 1. `ColumnsWritten` would contain `1` always while reading above blocks irrespective of whether Name1 or Name2 has been used as column name at the time of write operations.
When `Expression` and/or `Predicates` has columns (either Name1 or Name2) specified in it, both would be resolved to the same field id `1` and passed to evaluation (strict, inclusive etc) process. Including data files containing values written using `Name1` and files containing values written using `Name2` for remaining scan process is not going to cause any problems. It does not really matter how it is being called outside as long as we are using the same field id internally during reads and writes.
Is my understanding correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Collect list of columns written in file [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14126:
URL: https://github.com/apache/iceberg/pull/14126#issuecomment-3672841327
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org