You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/08/29 11:10:49 UTC

[GitHub] [arrow] motybz commented on pull request #10955: ARROW-13650: [C++] Create dataset writer to encapsulate dataset writer logic

motybz commented on pull request #10955:
URL: https://github.com/apache/arrow/pull/10955#issuecomment-907773672


   Hi
   
   In my POV the user should control the in-memory size of the datasets since the reason to separate the data set is to control the memory consumption of distributed data-consuming systems (like Spark, Presto, etc...).
   Moreover, the size of the final file of the data set depends on:
   number of columns * column type (int8/16/32, string, float... )* number of lines * compression.
   
   I can't see any benefit of controlling the number of the lines (besides the effect on the file size)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org