You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Kyle Kavanagh (Jira)" <ji...@apache.org> on 2020/09/25 13:50:00 UTC

[jira] [Created] (ARROW-10093) [R] Add ability to opt-out of int64 -> int demotion

Kyle Kavanagh created ARROW-10093:
-------------------------------------

             Summary: [R] Add ability to opt-out of int64 -> int demotion
                 Key: ARROW-10093
                 URL: https://issues.apache.org/jira/browse/ARROW-10093
             Project: Apache Arrow
          Issue Type: New Feature
            Reporter: Kyle Kavanagh


Currently, if arrow detects that every value in an int64 column can fit in a 32bit int, it will downcast the column an set the type to integer instead of integer64.  Not having a mechanism to disable this optimization makes it tricky if you have many parallel processes (think HPC use case) performing the same calculation but potentially outputting different result values, some being >2^32 and others not.  When you go to collect the resulting feather files from the parallel processes, the types across the files may not line up.

Feature request is to provide an option to disable this demotion and maintain the source column type.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)