[jira] [Commented] (PARQUET-474) InputStream and RandomAccessdSource classes are not threadsafe

["Wes McKinney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334441#comment-15334441 ] 

Wes McKinney commented on PARQUET-474:
--------------------------------------

Took a look at this. Reading a range of bytes from a local file must be made an atomic operation so that we can lock the source while performing a seek then read. Currently we have code like

{code}
  source_->Seek(filesize - FOOTER_SIZE);
  int64_t bytes_read = source_->Read(FOOTER_SIZE, footer_buffer);
{code}

This can be made thread-safe by having an API like

{code}
source_->ReadAt(filesize - FOOTER_SIZE, FOOTER_SIZE, footer_buffer);
{code}

We already have

{code}
  std::shared_ptr<Buffer> ReadAt(int64_t pos, int64_t nbytes);
{code}

in which we can also block other threads (as needed) while performing the read.

The stream classes are slightly different rabbit hole -- to get the best performance we'd want to have a buffered stream that continues to buffer data from the source in a background thread (presently it is synchronous / on-demand buffering)

> InputStream and RandomAccessdSource classes are not threadsafe
> --------------------------------------------------------------
>
>                 Key: PARQUET-474
>                 URL: https://issues.apache.org/jira/browse/PARQUET-474
>             Project: Parquet
>          Issue Type: Bug
>          Components: parquet-cpp
>            Reporter: Wes McKinney
>            Assignee: Wes McKinney
>
> We need to ensure that files can be processed in multithreaded applications



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)