You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Krisztian Szucs (Jira)" <ji...@apache.org> on 2020/09/14 21:35:00 UTC

[jira] [Created] (ARROW-10006) [C++][Python] Do not collect python iterators if not necessary

Krisztian Szucs created ARROW-10006:
---------------------------------------

             Summary: [C++][Python] Do not collect python iterators if not necessary
                 Key: ARROW-10006
                 URL: https://issues.apache.org/jira/browse/ARROW-10006
             Project: Apache Arrow
          Issue Type: New Feature
          Components: Python
            Reporter: Krisztian Szucs


When converting python objects to arrow array currently we always collect the input to a sequence, but this may be memory consuming in certain cases. 

For unknown sized iterators we could consume and temporarily store the seen items during inference potentially improving both the conversion time and peak memory usage. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)