[jira] [Created] (ARROW-4688) [C++][Parquet] 16MB limit on (nested) column chunk prevents tuning row_group_size

["Remek Zajac (JIRA)" <ji...@apache.org>]

Remek Zajac created ARROW-4688:
----------------------------------

             Summary: [C++][Parquet] 16MB limit on (nested) column chunk prevents tuning row_group_size
                 Key: ARROW-4688
                 URL: https://issues.apache.org/jira/browse/ARROW-4688
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++
            Reporter: Remek Zajac


We working on parquet files that involve nested lists. At most they are multi-dimensional lists of simple types (never structs), but i understand, for Parquet, they're still nested columns and involve repetition levels. 

Some of these columns hold lists of rather large byte arrays (that dominate the overall size of the row). When we bump the `row_group_size` to above 16MB we see: 

 
{code:java}
File "pyarrow/_parquet.pyx", line 700, in pyarrow._parquet.ParquetReader.read_row_group
 File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs{code}
 

I conclude it's [this|https://github.com/apache/arrow/blob/master/cpp/src/parquet/arrow/reader.cc#L848] bit complaining:

 
{code:java}
template <typename ParquetType>
	Status PrimitiveImpl::WrapIntoListArray(Datum* inout_array) {
	if (descr_->max_repetition_level() == 0) {
	  // Flat, no action
	  return Status::OK();
	}
	
	std::shared_ptr<Array> flat_array;
	
	// ARROW-3762(wesm): If inout_array is a chunked array, we reject as this is
	// not yet implemented
	if (inout_array->kind() == Datum::CHUNKED_ARRAY) {
	  if (inout_array->chunked_array()->num_chunks() > 1) {
	    return Status::NotImplemented(
	      "Nested data conversions not implemented for "
	      "chunked array outputs");{code}
 

This appears to happen in the callstack of ColumnReader::ColumnReaderImpl::NextBatch 
and it appears to be provoked by [this|https://github.com/apache/arrow/blob/de84293d9c93fe721cd127f1a27acc94fe290f3f/cpp/src/parquet/arrow/record_reader.cc#L604] constant:
{code:java}
template <>     
void TypedRecordReader<ByteArrayType>::InitializeBuilder() {     
  // Maximum of 16MB chunks     
  constexpr int32_t kBinaryChunksize = 1 << 24;     
  DCHECK_EQ(descr_->physical_type(), Type::BYTE_ARRAY);       
  builder_.reset(
    new::arrow::internal::ChunkedBinaryBuilder(kBinaryChunksize, pool_));  }   {code}
Which appears to imply that the column chunk data, if larger than kBinaryChunksize (hardcoded to 16MB), is returned as a Datum::CHUNKED_ARRAY of more than one (16MB) chunks. Which ultimatelly leads to the Status::NotImplemented error.

We have no influence over what data we ingest, we have some influence in how we flatten it and we need to tune our row_group_size to something sensibly larger than 16MB. 

We have see no obvious workaround for this and so we need to ask (1) if the above diagnosis appears to correct (2) do people see any sensible workarounds (3) is there an imminent intention to fix this in the Arrow community and if not, how difficult would it be to fix this (in case we can afford helping)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)