[jira] [Updated] (ARROW-7252) [Rust] [Parquet] Reading UTF-8/JSON/ENUM field results in a lot of vec allocation

["Neville Dipale (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neville Dipale updated ARROW-7252:
----------------------------------
    Fix Version/s: 4.0.0

> [Rust] [Parquet] Reading UTF-8/JSON/ENUM field results in a lot of vec allocation
> ---------------------------------------------------------------------------------
>
>                 Key: ARROW-7252
>                 URL: https://issues.apache.org/jira/browse/ARROW-7252
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Rust
>            Reporter: Wong Shek Hei
>            Priority: Minor
>             Fix For: 4.0.0
>
>
> While reading a very large parquet file with basically all string fields was very slow(430MB gzipped), after profiling with osx instruments, I noticed that a lot of time is spent in "convert_byte_array", in particular, "reserving" and allocating Vec::with_capacity, which is done before String::from_utf8_unchecked.
> It seems like using String as the underlying storage is causing this(String uses Vec<u8> for its underlying storage), this also requires copying from slice to vec.
> "Field::Str<String>" is a pub enum so I am not sure how "refactorable" is the String part, for example, converting it into a &str(we can perhaps then defer the conversion from &[u8] to Vec<u8> until the user really needs a String)
> But of course, changing it to &str can result in quite a bit of interface changes... So I am wondering if there are already some plans or solution on the way to improve the handling of the "Field::Str" case?
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)