You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "ezyang (via GitHub)" <gi...@apache.org> on 2023/09/02 02:01:41 UTC
[GitHub] [arrow] ezyang commented on pull request #37166: GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType
ezyang commented on PR #37166:
URL: https://github.com/apache/arrow/pull/37166#issuecomment-1703643362
This has been a contentious point for us internally. CC'ing @jbschlosser @cpuhrsch
If you want a generic container format that can represent anything that morally is a list of regular dense tensors, the most expressive way to do it is store individual strides per tensor. This is particularly useful if you have two jagged dimensions (e.g., list of varying size images on both height and width); if you store sizes/strides per batch this all just works fine.
However, it's a bit of a pain to store all the metadata, because most of the time you're not taking full advantage of all the expressivity; something a bit more compact will be more efficient, both in terms of memory usage and in terms of algorithmic efficiency. For example, it's common to have only one ragged dimension, so you have batch x ragged x some number of dense dimensions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org