You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "irshadcc (via GitHub)" <gi...@apache.org> on 2023/11/21 03:00:00 UTC
[I] Core : Optimise manifest evaluation for tables with large number of columns [iceberg]
irshadcc opened a new issue, #9118:
URL: https://github.com/apache/iceberg/issues/9118
### Feature Request / Improvement
During the snapshot commit process of MergingSnapshotProducer, Iceberg tries to merge the manifest files to increase the planning performance. During operations like overwrite, MergingSnapshotProducer finds the matching manifest files by deleteExpression and rewrites the manifests by filtering out the deleted manifest entries.
While finding the matching manifests by deleteExpression, we found that the Schema is created every time the expression needs to be bound. This has proven very expensive when there are large number of manifest files (~25,000 manifest files) for a super wide table (~35,000 columns).
For optimising the manifest evaluation, we can add a method called asSchema() in the StructType class to avoid creating the new Schema every time the filter needs to be
After caching the Schema for the StructType, we can avoid creating new schema while we try to bind the expression.
The optimisation reduced the manifest evaluation time from 06 minutes 22 seconds to 13 seconds.
### Query engine
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core : Optimise manifest evaluation for tables with large number of columns (30x faster) [iceberg]
Posted by "irshadcc (via GitHub)" <gi...@apache.org>.
irshadcc closed issue #9118: Core : Optimise manifest evaluation for tables with large number of columns (30x faster)
URL: https://github.com/apache/iceberg/issues/9118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core : Optimise manifest evaluation for tables with large number of columns [iceberg]
Posted by "irshadcc (via GitHub)" <gi...@apache.org>.
irshadcc commented on issue #9118:
URL: https://github.com/apache/iceberg/issues/9118#issuecomment-1825260470
After the optimisation, the time taken to evaluate 25,000 manifest files with schema containing 35,000 columns improved from 6 minutes 22 seconds to 13 seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core : Optimise manifest evaluation for tables with large number of columns [iceberg]
Posted by "irshadcc (via GitHub)" <gi...@apache.org>.
irshadcc commented on issue #9118:
URL: https://github.com/apache/iceberg/issues/9118#issuecomment-1825680468
Pull request #9147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org