[jira] [Created] (IMPALA-5307) Consider always copying-out Disk I/O buffers instead of attaching to RowBatches

["Tim Armstrong (JIRA)" <ji...@apache.org>]

Tim Armstrong created IMPALA-5307:
-------------------------------------

             Summary: Consider always copying-out Disk I/O buffers instead of attaching to RowBatches
                 Key: IMPALA-5307
                 URL: https://issues.apache.org/jira/browse/IMPALA-5307
             Project: IMPALA
          Issue Type: Sub-task
          Components: Backend
            Reporter: Tim Armstrong
            Assignee: Tim Armstrong


IMPALA-4835 would be greatly simplified if we don't have to attach disk I/O buffers to RowBatches and handle the resultant complexity.

Disk I/O buffers currently need to be attached to RowBatches if the row batches directly reference var-len data in the buffer. The cases when this can occur are as follows:
* The column being read contains strings
* The string data is not dictionary encoded in Parquet (since we copy out the dictionary data in Parquet)
* The string data is not compressed with a general-purpose compression algorithm (GZip, snappy, etc).

This includes the following cases: plain-encoded strings in uncompressed Parquet; any strings in uncompressed text, RCFile, Avro, or sequence file.

In those cases the copy avoidance could provide some performance benefits. However it's unclear that any of those file formats are/should be used in performance-critical use cases, because the storage density of uncompressed strings is almost always terrible.

We should evaluate the performance impact of the additional copies, but I suspect that it is not severe and does not impact any important use cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)