You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "zbs (via GitHub)" <gi...@apache.org> on 2025/02/27 05:22:16 UTC
[I] [C++] Writing batches to parquet batch-by-batch triggering OOM [arrow]
zbs opened a new issue, #45638:
URL: https://github.com/apache/arrow/issues/45638
### Describe the usage question you have. Please include as many useful details as possible.
I created a ParquetWriter class that initializes a schema/writer/outstream and writes to a parquet file one batch at a time; however, it seems to be OOMing and for the life of me I can not figure out why. I've experimented with different chunk sizes, calling flush after each write, but to no avail, and cannot find any documentation that indicates I'm doing anything seriously wrong. Further, I used valgrind and it did not find any memory leaks.
Note this is using arrow 9.0.0, so it's considerably behind, but I'm hoping the solution doesn't need to take that into account too much
```
class ParquetFileWriter {
public:
ParquetFileWriter(
const std::string & filename,
parquet::WriterProperties::Builder & writerPropertiesBuilder );
void initSchema( const std::shared_ptr< arrow::Schema > & schema ) override;
void writeBatch( const arrow::RecordBatch & batch ) override;
void poll() override;
private:
std::shared_ptr< arrow::io::OutputStream > outstream_;
std::unique_ptr< parquet::arrow::FileWriter > parquetWriter_;
std::shared_ptr< parquet::WriterProperties > writerProperties_;
};
ParquetFileWriter::ParquetFileWriter(
const std::string & filename,
parquet::WriterProperties::Builder & writerPropertiesBuilder )
: outstream_( arrow::io::FileOutputStream::Open( filename, false ).ValueOrDie() ) {
writerPropertiesBuilder.compression(parquet::Compression::ZSTD);
writerProperties_ = writerPropertiesBuilder.build();
}
void
ParquetFileWriter::initSchema( const std::shared_ptr< arrow::Schema > & schema ) {
auto status = parquet::arrow::FileWriter::Open(*schema, arrow::default_memory_pool(), outstream_, writerProperties_, &parquetWriter_);
VERIFY(status.ok(), "Failed to open Parquet file writer: %s", status.ToString().c_str());
}
void
ParquetFileWriter::writeBatch( const arrow::RecordBatch & batch ) {
check( parquetWriter_ != nullptr, "Parquet writer is not initialized." );
std::vector<std::shared_ptr<arrow::RecordBatch>> batches = { arrow::RecordBatch::Make(batch.schema(), batch.num_rows(), batch.columns()) };
auto table = arrow::Table::FromRecordBatches(batch.schema(), batches);
const arrow::Status status = parquetWriter_->WriteTable(**table, batch.num_rows());
check( status.ok(), "Parquet batch write failed: %s", status.ToString().c_str() );
check( flushStatus.ok(), "Flush status failed: %s", status.ToString().c_str() );
}
```
In the above code, `writeBatch` might be called thousands of times.
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Writing batches to parquet batch-by-batch triggering OOM [arrow]
Posted by "zbs (via GitHub)" <gi...@apache.org>.
zbs commented on issue #45638:
URL: https://github.com/apache/arrow/issues/45638#issuecomment-2689678138
> From the footprint above, it seems that metadata of written row groups consumes 16.25% of the total memory. Is something else dominating the memory consumption?
AFAICT nothing is dominating it -- I will try to increase the input subset used and run again. It's possible with my subsetting I've reduced the severity of the bug.
> Did you write a large number of row groups and columns?
Yes, dozens of files each with around 10K batches, totaling 10s of millions of rows. In many cases the batches contain 1 row. I disabled dictionary and statistics and will report back.
Re: better APIs, our entire environment needs to upgrade at the same time, so when that happens in a couple months, and this is still not resolved, hopefully these APIs will solve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Writing batches to parquet batch-by-batch triggering OOM [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #45638:
URL: https://github.com/apache/arrow/issues/45638#issuecomment-2686971628
What is the number of columns and number of rows in a single record batch? How many record batches do you write when OOM? Is it possible to profile the memory footprint?
Could you also try to bump your Arrow version and use `parquet::arrow::FileWriter::WriteRecordBatch`: https://github.com/apache/arrow/blob/main/cpp/src/parquet/arrow/writer.h#L132?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Writing batches to parquet batch-by-batch triggering OOM [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #45638: [C++] Writing batches to parquet batch-by-batch triggering OOM
URL: https://github.com/apache/arrow/issues/45638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Writing batches to parquet batch-by-batch triggering OOM [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #45638:
URL: https://github.com/apache/arrow/issues/45638#issuecomment-3996888310
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this usage question has evolved into a feature request or docs update, please remove the 'Type: usage' label and add the 'Type: enhancement' label instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Writing batches to parquet batch-by-batch triggering OOM [arrow]
Posted by "zbs (via GitHub)" <gi...@apache.org>.
zbs commented on issue #45638:
URL: https://github.com/apache/arrow/issues/45638#issuecomment-2688484841
I'll post updates here as I run diagnostics. The valgrind/massif run was not able to complete with the full input, so I ran it on a subset of the original input, and found this:
```
->16.25% (524,906,816B) 0x80EF7A3: std::vector<parquet::format::ColumnChunk, std::allocator<parquet::format::ColumnChunk> >::operator=(std::vector<parquet::format::ColumnChunk, std::allocator<parquet::format::ColumnChunk> > const&) [clone .isra.0]
| ->16.25% (524,906,816B) 0x80F341D: parquet::format::RowGroup::RowGroup(parquet::format::RowGroup const&)
| | ->16.25% (524,887,216B) 0x80D36BA: void std::vector<parquet::format::RowGroup, std::allocator<parquet::format::RowGroup> >::_M_realloc_insert<>(__gnu_cxx::__normal_iterator<parquet::format::RowGroup*, std::vector<parquet::format::RowGroup, std::allocator<parquet::format::RowGroup> > >)
| | | ->16.25% (524,887,216B) 0x80CC3B3: parquet::FileMetaDataBuilder::AppendRowGroup()
| | | ->16.25% (524,887,216B) 0x80BF59C: parquet::FileSerializer::AppendRowGroup(bool)
| | | ->16.24% (524,797,056B) 0x7FEC1A7: parquet::arrow::FileWriterImpl::WriteTable(arrow::Table const&, long)
| | | | ->16.24% (524,797,056B) 0x46F829F: Namespace::ParquetFileWriter::writeBatch(arrow::RecordBatch const&) (ArrowWriter.cpp:269)
<----------------------------snip----------------------------------->
| | | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| | | |
| | | ->00.00% (90,160B) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->00.00% (19,600B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Writing batches to parquet batch-by-batch triggering OOM [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #45638:
URL: https://github.com/apache/arrow/issues/45638#issuecomment-2689516620
From the footprint above, it seems that metadata of written row groups consumes 16.25% of the total memory. Is something else dominating the memory consumption?
Did you write a large number of row groups and columns? For `WriteTable`, each call will produce at least one row group. Too many small row groups may result in equivalent number of row group metadata: https://github.com/apache/arrow/blob/main/cpp/src/parquet/metadata.cc#L2059. In the `format::RowGroup` metadata, every column needs to store its column name and column statistics. If you have a lot of string typed columns, the memory consumption is significant. You may try to set `WriterProperties::disable_statistics()` to disable it to see if it helps. I still suggest that `FileWriter::NewBufferedRowGroup()` and `FileWriter::WriteRecordBatch()` are better APIs for this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org