[jira] [Assigned] (ORC-1116) Csv-import tool exported field become empty

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ORC-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun reassigned ORC-1116:
----------------------------------

    Assignee: kyle

> Csv-import tool exported field become empty
> -------------------------------------------
>
>                 Key: ORC-1116
>                 URL: https://issues.apache.org/jira/browse/ORC-1116
>             Project: ORC
>          Issue Type: Bug
>          Components: tools
>    Affects Versions: 1.7.3
>            Reporter: kyle
>            Assignee: kyle
>            Priority: Minor
>             Fix For: 1.7.4
>
>         Attachments: CSVFileImport.dif
>
>
> When exporting a orc file with schema like "struct<a:string,b:binary>", if the data in column "b" has very long bytes (over 4MB), the process could segmentation fault or exported data in column "a" becomes empty string.
> Here is me trying to explain the code, maybe totally not correct, please bear with me.
> Following the code in CSVFileImport.cc, when writing to a orc file, all string type columns is using one databuffer inside function fillStringValues(). When one data length is larger than the buffer, the buffer will be resized. The resize() operation will cause all references and iterators into buffer.data() become invalid. 
> In this case, when field "a" finished writing data into buffer, field "b" begin writing will resize the buffer, invalidate previous buffer.data(), so field "a"'s stringBatch pointing to buffer.data() is no longer valid.
> A workaround could use different databuffers for each string type column, however requires allocating 4MB memory each. (As the attached file) Or let all previous stringBatch re-points to new databuffer's address.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)