[jira] [Created] (ORC-1116) Csv-import tool exported field become empty

["kyle (Jira)" <ji...@apache.org>]

kyle created ORC-1116:
-------------------------

             Summary: Csv-import tool exported field become empty
                 Key: ORC-1116
                 URL: https://issues.apache.org/jira/browse/ORC-1116
             Project: ORC
          Issue Type: Bug
          Components: tools
    Affects Versions: 1.7.3
            Reporter: kyle
         Attachments: CSVFileImport.dif

When exporting a orc file with schema like "struct<a:string,b:binary>", if the data in column "b" has very long bytes (over 4MB), the process could segmentation fault or exported data in column "a" becomes empty string.

Here is me trying to explain the code, maybe totally not correct, please bear with me.

Following the code in CSVFileImport.cc, when writing to a orc file, all string type columns is using one databuffer inside function fillStringValues(). When one data length is larger than the buffer, the buffer will be resized. The resize() operation will cause all references and iterators into buffer.data() become invalid. 

In this case, when field "a" finished writing data into buffer, field "b" begin writing will resize the buffer, invalidate previous buffer.data(), so field "a"'s stringBatch pointing to buffer.data() is no longer valid.

A workaround could use different databuffers for each string type column, however requires allocating 4MB memory each. (As the attached file) Or let all previous stringBatch re-points to new databuffer's address.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)