[GitHub] [arrow-rs] alamb commented on issue #4811: Stateless Row Conversion

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on issue #4811:
URL: https://github.com/apache/arrow-rs/issues/4811#issuecomment-1717787949

   I personally think this proposal is very compelling because using high cardinality dictionaries has caused us significant pain in IOx due to the memory accumulated in the row converters. We have several non trivial PRs to properly account for and try to avoid using dictionary encoding for these cases -- see https://github.com/apache/arrow-datafusion/pull/7130 and  https://github.com/apache/arrow-datafusion/pull/7401 for example
   
   Also, a stateful row converter has caused other operations like https://github.com/apache/arrow-datafusion/pull/7379 from @wiedld become significantly more complicatated as it can't be shared between threads
   
   Thus, in my mind the ideal outcome would be that we can remove the stateful row conversion and minimize the performance penalty for relatively low cardinality dictionaries.  I am hopeful that @tustvold 's ideas to make string encoding more efficient (e.g. https://github.com/apache/arrow-rs/issues/4812) could be part of this answer
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org