You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "irshadcc (via GitHub)" <gi...@apache.org> on 2023/11/21 03:00:00 UTC

[I] Core : Optimise manifest evaluation for tables with large number of columns [iceberg]

irshadcc opened a new issue, #9118:
URL: https://github.com/apache/iceberg/issues/9118

   ### Feature Request / Improvement
   
   During the snapshot commit process of MergingSnapshotProducer, Iceberg tries to merge the manifest files to increase the planning performance. During operations like overwrite, MergingSnapshotProducer finds the matching manifest files by deleteExpression and rewrites the manifests by filtering out the deleted manifest entries. 
   
   While finding the matching manifests by deleteExpression, we found that the Schema is created every time the expression needs to be bound. This has proven very expensive when there are large number of manifest files (~25,000 manifest files) for a super wide table (~35,000 columns).
   
   For optimising the manifest evaluation, we can add a method called asSchema() in the StructType class to avoid creating the new Schema every time the filter needs to be 
   After caching the Schema for the StructType, we can avoid creating new schema while we try to bind the expression. 
   
   The optimisation reduced the manifest evaluation time from 06 minutes 22 seconds to 13 seconds.
   
   ### Query engine
   
   None


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Core : Optimise manifest evaluation for tables with large number of columns (30x faster) [iceberg]

Posted by "irshadcc (via GitHub)" <gi...@apache.org>.
irshadcc closed issue #9118: Core : Optimise manifest evaluation for tables with large number of columns (30x faster)
URL: https://github.com/apache/iceberg/issues/9118


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Core : Optimise manifest evaluation for tables with large number of columns [iceberg]

Posted by "irshadcc (via GitHub)" <gi...@apache.org>.
irshadcc commented on issue #9118:
URL: https://github.com/apache/iceberg/issues/9118#issuecomment-1825260470

   After the optimisation, the time taken to evaluate 25,000 manifest files with schema containing 35,000 columns improved from 6 minutes 22 seconds to 13 seconds. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Core : Optimise manifest evaluation for tables with large number of columns [iceberg]

Posted by "irshadcc (via GitHub)" <gi...@apache.org>.
irshadcc commented on issue #9118:
URL: https://github.com/apache/iceberg/issues/9118#issuecomment-1825680468

   Pull request #9147 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org