You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/12/13 06:07:27 UTC
[GitHub] [iceberg] rbalamohan opened a new pull request, #6417: Reuse existing parquet reader in ReadConf (6416)
rbalamohan opened a new pull request, #6417:
URL: https://github.com/apache/iceberg/pull/6417
https://github.com/apache/iceberg/issues/6416
ReadConf creates new file reader for reading "generateOffsetToStartPos". Though it will be around 30-50ms call, this quickly increases overall runtime in workloads having lot of positional delete files. Creating this as a placeholder ticket to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Reuse existing parquet reader in ReadConf (6416) [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #6417: Reuse existing parquet reader in ReadConf (6416)
URL: https://github.com/apache/iceberg/pull/6417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #6417: Reuse existing parquet reader in ReadConf (6416)
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #6417:
URL: https://github.com/apache/iceberg/pull/6417#discussion_r1051684313
##########
parquet/src/main/java/org/apache/iceberg/parquet/ReadConf.java:
##########
@@ -185,21 +184,16 @@ private Map<Long, Long> generateOffsetToStartPos(Schema schema) {
return null;
}
- try (ParquetFileReader fileReader = newReader(file, ParquetReadOptions.builder().build())) {
Review Comment:
Unfortunately, this isn't correct. The options that are passed into ReadConf will include the split's range of bytes in the file, and Parquet will use that to filter metadata before returning it, so `fileReader.getRowGroups()` will be different. We need to avoid passing down that filter.
I think it's debatable whether we need to actually filter metadata. It seems like a silly optimization to me, when we will generally have very few row groups in a file. I think it was an optimization for Parquet doing some pretty dumb things in the past.
If you want, we can [remove the range pushdown](https://github.com/apache/iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L1027) and continue with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Reuse existing parquet reader in ReadConf (6416) [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6417:
URL: https://github.com/apache/iceberg/pull/6417#issuecomment-2307946079
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Reuse existing parquet reader in ReadConf (6416) [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6417:
URL: https://github.com/apache/iceberg/pull/6417#issuecomment-2322625490
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org