You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/10/01 05:38:22 UTC
[GitHub] [iceberg] szehon-ho opened a new issue #3211: Add Expression filter to RewriteManifest to allow predicate pushdown
szehon-ho opened a new issue #3211:
URL: https://github.com/apache/iceberg/issues/3211
Some tables may end up in a bad state with tens of thousands of ManifestFiles. RewriteManifests, even with a very restrictive predicate on the Manifests, ends up not being able to finish in a reasonable time. (Imagine the table is the target of a Flink ingest job, for example).
This is because all Manifests have to be read in order to evaluate the Predicate.
So, propose to add a rewriteIf(Expression filter) on the RewriteManifest API as an alternate to rewriteIf(Predicate<ManifestFIle>). This is then able to push down into the ManifestGroup and reduce the ManifestFiles to read, to be compacted.
Also, it is more user-friendly. Trying to filter ManifestFile with the predicate using the ManifestFile.PartitionFieldSummary() is not user-friendly at all as it's upper and lower bounds are in byte format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #3211: Add Expression filter to RewriteManifest to allow predicate pushdown
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #3211:
URL: https://github.com/apache/iceberg/issues/3211#issuecomment-931918807
@rdblue @RussellSpitzer wondering if this sounds reasonable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #3211: Add Expression filter to RewriteManifest to allow predicate pushdown
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #3211:
URL: https://github.com/apache/iceberg/issues/3211#issuecomment-931960218
Never mind, it looks like it is doing the filter properly, sorry for noise, need further investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho closed issue #3211: Add Expression filter to RewriteManifest to allow predicate pushdown
Posted by GitBox <gi...@apache.org>.
szehon-ho closed issue #3211:
URL: https://github.com/apache/iceberg/issues/3211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho removed a comment on issue #3211: Add Expression filter to RewriteManifest to allow predicate pushdown
Posted by GitBox <gi...@apache.org>.
szehon-ho removed a comment on issue #3211:
URL: https://github.com/apache/iceberg/issues/3211#issuecomment-931918807
@rdblue @RussellSpitzer wondering if this sounds reasonable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org