[jira] [Created] (ARROW-5129) Column writer bug: check dictionary encoder when adding a new data page

["Ivan Sadikov (JIRA)" <ji...@apache.org>]

Ivan Sadikov created ARROW-5129:
-----------------------------------

             Summary: Column writer bug: check dictionary encoder when adding a new data page
                 Key: ARROW-5129
                 URL: https://issues.apache.org/jira/browse/ARROW-5129
             Project: Apache Arrow
          Issue Type: Bug
          Components: Rust
         Environment: N/A
            Reporter: Ivan Sadikov


As part of my weekly routine, I glanced over code in Parquet column writer and found that the way we check when to add a new data page is buggy. The idea is checking the current encoder and deciding if we have written enough bytes for a page to construct. The problem is that we only check value encoder, regardless whether or not dictionary encoder is enabled. 

Here is how we do it now: actual check (https://github.com/apache/arrow/blob/master/rust/parquet/src/column/writer.rs#L378) and the buggy function (https://github.com/apache/arrow/blob/master/rust/parquet/src/column/writer.rs#L423). 

In the case of sparse column and dictionary  encoder we would write a single data page, even though we would have accumulated a large enough number of bytes for more than one page in encoder (value encoder will be empty, so it will always less than constant limit).

I forgot that parquet-cpp has `current_encoder` as either value encoder or dictionary encoder (https://github.com/apache/parquet-cpp/blob/master/src/parquet/column_writer.cc#L544), but in parquet-rs we have them separate.

So the fix could be something like this:
{code}
/// Returns true if there is enough data for a data page, false otherwise.
#[inline]
fn should_add_data_page(&self) -> bool {
  match self.dict_encoder {
    Some(ref encoder) => {
      encoder.estimated_data_encoded_size() >= self.props.data_pagesize_limit()
    },
    None => {
      self.encoder.estimated_data_encoded_size() >= self.props.data_pagesize_limit()
    }
  }
}
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)