You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "rbalamohan (via GitHub)" <gi...@apache.org> on 2023/02/02 09:09:27 UTC
[GitHub] [iceberg] rbalamohan opened a new issue, #6726: Dynamic partition pruning filters should be applied before invoking Table::planTasks in IcebergInputFormat
rbalamohan opened a new issue, #6726:
URL: https://github.com/apache/iceberg/issues/6726
### Apache Iceberg version
1.0.0
### Query engine
Hive
### Please describe the bug 🐞
Context:
======
Split computation in iceberg takes 3-5 seconds (in certain queries more than that); This is significantly higher when compared to regular tables, which were in the order of 100s of milliseconds. BI kind of query latencies are impacted due to split computation runtimes.
Issue:
=====
Dynamic partition filtering is a powerful optimization to reduce the number of partitions being scanned at runtime. In hive, it creates the filter at runtime and pushes it to downstream vertices at runtime.
In iceberg, split computation in IcebergInputFormat happens via scan.planTasks. Filters if any are applied later. This causes metadata for all partitions to be read and then applies the filter causing computatioin delays. It will be good to figure out if the filtering can be applied earlier or in optimal manner.
https://github.com/apache/iceberg/blob/dbb8a404f6632a55acb36e949f0e7b84b643cede/mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java#L147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6726: Dynamic partition pruning filters should be applied before invoking Table::planTasks in IcebergInputFormat
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6726:
URL: https://github.com/apache/iceberg/issues/6726#issuecomment-1661270816
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Dynamic partition pruning filters should be applied before invoking Table::planTasks in IcebergInputFormat [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6726: Dynamic partition pruning filters should be applied before invoking Table::planTasks in IcebergInputFormat
URL: https://github.com/apache/iceberg/issues/6726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Dynamic partition pruning filters should be applied before invoking Table::planTasks in IcebergInputFormat [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6726:
URL: https://github.com/apache/iceberg/issues/6726#issuecomment-1890802436
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org