You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@orc.apache.org by "zabetak (via GitHub)" <gi...@apache.org> on 2023/02/22 11:49:44 UTC

[GitHub] [orc] zabetak commented on issue #1404: Support read large statistics exceed 2GB

zabetak commented on issue #1404:
URL: https://github.com/apache/orc/issues/1404#issuecomment-1439889316

   At first glance 2GB of metadata seems big especially when considering the toy example that I made in ORC-1361. However, if you have a 500GB ORC file then 2GB of metadata does not appear too big anymore so things are relative.
   
   Are there limitations on the maximum size of an ORC file? Do we support such kind of use-cases?
   
   If we add a limit while writing (which by the way I also suggested in https://github.com/protocolbuffers/protobuf/issues/11729) then we should define what happens when the limit is exceeded:
   * drop all metadata
   * fail the entire write
   * ignore metadata over the limit (keep partial metadata)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@orc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org