You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2024/05/21 15:27:48 UTC
[PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
pitrou opened a new pull request, #41761:
URL: https://github.com/apache/arrow/pull/41761
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
### What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608645086
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
> Did you check the data? It's tiny (one INT32 value per column chunk). The entire file will consist of metadata overhead.
I'll download one. I think we'd better get "concrete" size of footer, but we can choose current impl if you consist
( Besides, footer read by default would be 64KiB, if the footer is greater than 64KiB, a more around of read and parse might be run here)
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
> Did you check the data? It's tiny (one INT32 value per column chunk). The entire file will consist of metadata overhead.
I'll download one. I think we'd better get "concrete" size of footer, but we can choose current impl if you insist
( Besides, footer read by default would be 64KiB, if the footer is greater than 64KiB, a more around of read and parse might be run here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41761:
URL: https://github.com/apache/arrow/pull/41761#issuecomment-2122894466
Benchmark results here. We see that performance is _O(num_columns * num_row_groups)_.
```
-------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-------------------------------------------------------------------------------------------------------------
WriteMetadata/num_columns:1/num_row_groups:1 11493 ns 11491 ns 61331 file_size=459 items_per_second=87.0256k/s
WriteMetadata/num_columns:1/num_row_groups:100 820026 ns 819815 ns 854 file_size=37.383k items_per_second=1.21979k/s
WriteMetadata/num_columns:1/num_row_groups:1000 8024208 ns 8022519 ns 87 file_size=374.885k items_per_second=124.649/s
WriteMetadata/num_columns:10/num_row_groups:1 98586 ns 98558 ns 7083 file_size=3.762k items_per_second=10.1463k/s
WriteMetadata/num_columns:10/num_row_groups:100 7816090 ns 7814200 ns 89 file_size=358.835k items_per_second=127.972/s
WriteMetadata/num_columns:10/num_row_groups:1000 79490918 ns 79462535 ns 8 file_size=3.614M items_per_second=12.5845/s
WriteMetadata/num_columns:100/num_row_groups:1 932833 ns 932560 ns 759 file_size=37.352k items_per_second=1.07232k/s
WriteMetadata/num_columns:100/num_row_groups:100 78799934 ns 78771226 ns 9 file_size=3.61693M items_per_second=12.695/s
WriteMetadata/num_columns:100/num_row_groups:1000 857600506 ns 857330657 ns 1 file_size=36.2887M items_per_second=1.16641/s
WriteMetadata/num_columns:1000/num_row_groups:1 9051274 ns 9049407 ns 77 file_size=376.655k items_per_second=110.504/s
WriteMetadata/num_columns:1000/num_row_groups:100 827747343 ns 827468643 ns 1 file_size=36.4815M items_per_second=1.20851/s
WriteMetadata/num_columns:10000/num_row_groups:1 95165920 ns 95125167 ns 7 file_size=3.82213M items_per_second=10.5125/s
WriteMetadata/num_columns:10000/num_row_groups:100 8698273757 ns 8693696946 ns 1 file_size=369.089M items_per_second=0.115026/s
ReadMetadata/num_columns:1/num_row_groups:1 3767 ns 3766 ns 185550 file_size=459 items_per_second=265.553k/s
ReadMetadata/num_columns:1/num_row_groups:100 87250 ns 87235 ns 8004 file_size=37.383k items_per_second=11.4633k/s
ReadMetadata/num_columns:1/num_row_groups:1000 831546 ns 831380 ns 842 file_size=374.885k items_per_second=1.20282k/s
ReadMetadata/num_columns:10/num_row_groups:1 19477 ns 19474 ns 35220 file_size=3.762k items_per_second=51.3513k/s
ReadMetadata/num_columns:10/num_row_groups:100 698405 ns 698268 ns 994 file_size=358.835k items_per_second=1.43211k/s
ReadMetadata/num_columns:10/num_row_groups:1000 6841245 ns 6839685 ns 102 file_size=3.614M items_per_second=146.206/s
ReadMetadata/num_columns:100/num_row_groups:1 174932 ns 174898 ns 3979 file_size=37.352k items_per_second=5.71763k/s
ReadMetadata/num_columns:100/num_row_groups:100 6640500 ns 6638581 ns 105 file_size=3.61693M items_per_second=150.635/s
ReadMetadata/num_columns:100/num_row_groups:1000 75471970 ns 75433100 ns 9 file_size=36.2887M items_per_second=13.2568/s
ReadMetadata/num_columns:1000/num_row_groups:1 1671059 ns 1670522 ns 421 file_size=376.655k items_per_second=598.615/s
ReadMetadata/num_columns:1000/num_row_groups:100 74756894 ns 74713295 ns 9 file_size=36.4815M items_per_second=13.3845/s
ReadMetadata/num_columns:10000/num_row_groups:1 17139819 ns 17132091 ns 41 file_size=3.82213M items_per_second=58.37/s
ReadMetadata/num_columns:10000/num_row_groups:100 774363229 ns 773118912 ns 1 file_size=369.089M items_per_second=1.29346/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608634164
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
File has been write with different batches with multiple columns. The file_size might also changed if `encoding` changed. Instead of batch-size, shouldn't we focus more on file metadata thrift size / runtrip, since no "data" is being read, and here we only benchmark reading metadata?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608618250
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
> The stats are already encoded so I think it is pretty trivial. But may be different if it is BYTE_ARRAY type with large strings.
That'd be valuable, but here we want to benchmark metadata cost in isolation of other factors?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "m4rs-mt (via GitHub)" <gi...@apache.org>.
m4rs-mt commented on PR #41761:
URL: https://github.com/apache/arrow/pull/41761#issuecomment-2124639700
@pitrou, thank you very much for your work on these benchmarks. They actually confirm the observation we made months ago. Great job! 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608603006
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
(besides, should we later add benchmark for byte-array statistics decoding ?)
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
Should we later add benchmark for drop statistics? ( this is useful in some case )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608616880
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
And we can add it in further patches, this could be a good start
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41761:
URL: https://github.com/apache/arrow/pull/41761#issuecomment-2124724552
I'll merge once CI is green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608606070
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
What do you mean with "drop statistics"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1609158213
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
I think the concern from @mapleFU is that the write and read paths in this benchmark are not identical. On the read path, only file footer is loaded. However, there are more work done on the write path (e.g. encoding and writing page headers).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #41761:
URL: https://github.com/apache/arrow/pull/41761#issuecomment-2125255008
After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit f3d46398d3c81d9575ffd77ce3b86d4b993a4888.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/25290175820) has more details. It also includes information about 12 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608679727
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
"Metadata" is not only the footer, it's all the Thrift overhead + other signaling : page headers, statistics...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1609450195
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
Ah, I see, that's a good point. I can try to rewrite the write benchmark so as to only measure the metadata write part of writing a file, though that's dependent on where exactly it happens in the Parquet API calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41761:
URL: https://github.com/apache/arrow/pull/41761#issuecomment-2124632845
Besides changing the benchmark names, I've fixed a bug where the benchmark was writing too many rows. Updated benchmark numbers:
```
------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------------------------
WriteFileMetadataAndData/num_columns:1/num_row_groups:1 11753 ns 11751 ns 60000 data_size=54 file_size=290 items_per_second=85.0979k/s
WriteFileMetadataAndData/num_columns:1/num_row_groups:100 826599 ns 826369 ns 845 data_size=5.4k file_size=20.486k items_per_second=1.21011k/s
WriteFileMetadataAndData/num_columns:1/num_row_groups:1000 8125279 ns 8123486 ns 86 data_size=54k file_size=207.687k items_per_second=123.1/s
WriteFileMetadataAndData/num_columns:10/num_row_groups:1 100357 ns 100329 ns 6982 data_size=540 file_size=2.11k items_per_second=9.96719k/s
WriteFileMetadataAndData/num_columns:10/num_row_groups:100 7913763 ns 7911870 ns 89 data_size=54k file_size=193.673k items_per_second=126.392/s
WriteFileMetadataAndData/num_columns:10/num_row_groups:1000 80283750 ns 80259596 ns 8 data_size=540k file_size=1.94392M items_per_second=12.4596/s
WriteFileMetadataAndData/num_columns:100/num_row_groups:1 937374 ns 937047 ns 745 data_size=5.4k file_size=20.694k items_per_second=1.06718k/s
WriteFileMetadataAndData/num_columns:100/num_row_groups:100 78617577 ns 78590896 ns 9 data_size=540k file_size=1.94729M items_per_second=12.7241/s
WriteFileMetadataAndData/num_columns:100/num_row_groups:1000 847927859 ns 847245720 ns 1 data_size=5.4M file_size=19.7673M items_per_second=1.1803/s
WriteFileMetadataAndData/num_columns:1000/num_row_groups:1 9169263 ns 9167280 ns 76 data_size=54k file_size=211.499k items_per_second=109.084/s
WriteFileMetadataAndData/num_columns:1000/num_row_groups:100 826783518 ns 826541970 ns 1 data_size=5.4M file_size=19.9623M items_per_second=1.20986/s
WriteFileMetadataAndData/num_columns:10000/num_row_groups:1 97516452 ns 97475073 ns 7 data_size=540k file_size=2.15317M items_per_second=10.259/s
WriteFileMetadataAndData/num_columns:10000/num_row_groups:100 8294650136 ns 8292024551 ns 1 data_size=54M file_size=201.916M items_per_second=0.120598/s
ReadFileMetadata/num_columns:1/num_row_groups:1 3669 ns 3668 ns 190349 data_size=54 file_size=290 items_per_second=272.625k/s
ReadFileMetadata/num_columns:1/num_row_groups:100 84829 ns 84814 ns 8322 data_size=5.4k file_size=20.486k items_per_second=11.7905k/s
ReadFileMetadata/num_columns:1/num_row_groups:1000 814708 ns 814559 ns 858 data_size=54k file_size=207.687k items_per_second=1.22766k/s
ReadFileMetadata/num_columns:10/num_row_groups:1 19247 ns 19243 ns 36787 data_size=540 file_size=2.11k items_per_second=51.9679k/s
ReadFileMetadata/num_columns:10/num_row_groups:100 690304 ns 690181 ns 1006 data_size=54k file_size=193.673k items_per_second=1.4489k/s
ReadFileMetadata/num_columns:10/num_row_groups:1000 6723370 ns 6721934 ns 105 data_size=540k file_size=1.94392M items_per_second=148.767/s
ReadFileMetadata/num_columns:100/num_row_groups:1 171196 ns 171161 ns 4077 data_size=5.4k file_size=20.694k items_per_second=5.84244k/s
ReadFileMetadata/num_columns:100/num_row_groups:100 6585073 ns 6583501 ns 106 data_size=540k file_size=1.94729M items_per_second=151.895/s
ReadFileMetadata/num_columns:100/num_row_groups:1000 75581479 ns 75544415 ns 9 data_size=5.4M file_size=19.7673M items_per_second=13.2372/s
ReadFileMetadata/num_columns:1000/num_row_groups:1 1631872 ns 1631517 ns 435 data_size=54k file_size=211.499k items_per_second=612.926/s
ReadFileMetadata/num_columns:1000/num_row_groups:100 75144953 ns 75104316 ns 9 data_size=5.4M file_size=19.9623M items_per_second=13.3148/s
ReadFileMetadata/num_columns:10000/num_row_groups:1 16866177 ns 16857406 ns 41 data_size=540k file_size=2.15317M items_per_second=59.3211/s
ReadFileMetadata/num_columns:10000/num_row_groups:100 780744719 ns 779370723 ns 1 data_size=54M file_size=201.916M items_per_second=1.28309/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608634164
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
File has been write with different batches with multiple columns. The file_size might also changed if `encoding` changed. Instead of batch-size, shouldn't we focus more on file metadata thrift size?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608661782
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
I also check the case for `num_columns:1`, the offset is `Footer offset: 287, size: 164`. Maybe footer is 1/2 - 1/4 size of file in this scenerio
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608687360
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
You're right
```
auto reader = ParquetFileReader::Open(source, props);
auto metadata = reader->metadata();
```
Any way, this only reads from footer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608679727
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
"Metadata" is not only the footer, it's all the Thrift overhead + other signaling : page headers, statistics, column chunk metadata...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41761:
URL: https://github.com/apache/arrow/pull/41761#issuecomment-2122891768
@mapleFU @emkornfield FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608606207
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
(decrypt can also be added later)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608606422
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
> (besides, should we later add benchmark for byte-array statistics decoding ?)
We can add benchmarks for whatever may be a performance bottleneck :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #41761:
URL: https://github.com/apache/arrow/pull/41761
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608607602
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
add `metadata()->SerializeToString().size()` here?
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
add `metadata()->SerializeToString().size()` as metadata size here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608609648
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
> What do you mean with "drop statistics"?
By default, parquet enable "statistics" for column, this will add a `Statistics` in column-chunk. We may add benchmark for droping the "statistics". And for ByteArray, this might be a bit long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608607310
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
The stats are already encoded so I think it is pretty trivial. But may be different if it is BYTE_ARRAY type with large strings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608634164
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
File has been write with different batches. The file_size might also changed if `encoding` changed. Instead of batch-size, shouldn't we focus more on file metadata thrift size?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608659675
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
After checking, I've generate the case for
* `ReadMetadata/num_columns:10000/num_row_groups:1`, the result is: `Footer offset: 2901570, size: 920550`. Footer occupies 900K and is 1/3 less than file size.
I think this might because some metadata has been "dual" storaged...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "corwinjoy (via GitHub)" <gi...@apache.org>.
corwinjoy commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1609209402
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
It's a good idea to also try this with the statistics dropped since these do have a big impact and have been heavily discussed. That is to say, run the numbers with `builder.disable_statistics()`. Based on previous profiling I have done in https://github.com/apache/arrow/issues/39676 this will cut the metadata read time by about 1/3. Still the main conclusion that the scaling is _O(num_columns * num_row_groups)_ should still hold. @pitrou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "corwinjoy (via GitHub)" <gi...@apache.org>.
corwinjoy commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1609210244
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
Review Comment:
Also, thanks a bunch for going to the effort to create these benchmarks! I think they are very helpful for the discussion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608631002
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
What additional information would it give?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1608640180
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
> File has been write with different batches with multiple columns.
Varying the number of columns is important here.
> The file_size might also changed if `encoding` changed.
Did you check the data? It's tiny (one INT32 value per column chunk). The entire file will consist of metadata overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41760: [C++][Parquet] Add file metadata read/write benchmark [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41761:
URL: https://github.com/apache/arrow/pull/41761#discussion_r1609475814
##########
cpp/src/parquet/metadata_benchmark.cc:
##########
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <sstream>
+
+#include <benchmark/benchmark.h>
+
+#include "arrow/buffer.h"
+#include "arrow/io/memory.h"
+#include "arrow/util/logging.h"
+
+#include "parquet/column_writer.h"
+#include "parquet/file_reader.h"
+#include "parquet/file_writer.h"
+#include "parquet/metadata.h"
+#include "parquet/platform.h"
+#include "parquet/schema.h"
+
+namespace parquet {
+
+using ::arrow::Buffer;
+using ::arrow::io::BufferOutputStream;
+using ::arrow::io::BufferReader;
+using schema::GroupNode;
+using schema::NodePtr;
+using schema::NodeVector;
+
+class MetadataBenchmark {
+ public:
+ explicit MetadataBenchmark(benchmark::State* state)
+ : MetadataBenchmark(static_cast<int>(state->range(0)),
+ static_cast<int>(state->range(1))) {}
+
+ MetadataBenchmark(int num_columns, int num_row_groups)
+ : num_columns_(num_columns), num_row_groups_(num_row_groups) {
+ NodeVector fields;
+ for (int i = 0; i < num_columns_; ++i) {
+ std::stringstream ss;
+ ss << "col" << i;
+ fields.push_back(parquet::schema::Int32(ss.str(), Repetition::REQUIRED));
+ }
+ schema_root_ = std::static_pointer_cast<GroupNode>(
+ GroupNode::Make("schema", Repetition::REQUIRED, fields));
+
+ WriterProperties::Builder prop_builder;
+ writer_properties_ =
+ prop_builder.version(ParquetVersion::PARQUET_2_6)->disable_dictionary()->build();
+ }
+
+ std::shared_ptr<BufferOutputStream> WriteFile() {
+ PARQUET_ASSIGN_OR_THROW(auto sink, BufferOutputStream::Create());
+
+ auto writer = ParquetFileWriter::Open(sink, schema_root_, writer_properties_);
+ std::vector<int32_t> int32_values(42, 1);
+ for (int rg = 0; rg < num_row_groups_; ++rg) {
+ auto row_group_writer = writer->AppendRowGroup();
+ for (int col = 0; col < num_columns_; ++col) {
+ auto col_writer = row_group_writer->NextColumn();
+ ARROW_CHECK_EQ(col_writer->type(), Type::INT32);
+ auto typed_col_writer = static_cast<Int32Writer*>(col_writer);
+ typed_col_writer->WriteBatch(
+ /*num_values=*/static_cast<int64_t>(int32_values.size()),
+ /*def_levels=*/nullptr, /*rep_levels=*/nullptr, int32_values.data());
+ typed_col_writer->Close();
+ }
+ row_group_writer->Close();
+ }
+ writer->Close();
+ return sink;
+ }
+
+ void ReadFile(std::shared_ptr<Buffer> contents) {
+ auto source = std::make_shared<BufferReader>(contents);
+ ReaderProperties props;
+ auto reader = ParquetFileReader::Open(source, props);
+ auto metadata = reader->metadata();
+ ARROW_CHECK_EQ(metadata->num_columns(), num_columns_);
+ ARROW_CHECK_EQ(metadata->num_row_groups(), num_row_groups_);
+ reader->Close();
+ }
+
+ private:
+ int num_columns_;
+ int num_row_groups_;
+ std::shared_ptr<GroupNode> schema_root_;
+ std::shared_ptr<WriterProperties> writer_properties_;
+};
+
+void WriteMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ bench->ArgNames({"num_columns", "num_row_groups"});
+
+ for (int num_columns : {1, 10, 100}) {
+ for (int num_row_groups : {1, 100, 1000}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+ /* For larger num_columns, restrict num_row_groups to small values
+ * to avoid blowing up benchmark execution time.
+ */
+ for (int num_columns : {1000, 10000}) {
+ for (int num_row_groups : {1, 100}) {
+ bench->Args({num_columns, num_row_groups});
+ }
+ }
+}
+
+void ReadMetadataSetArgs(benchmark::internal::Benchmark* bench) {
+ WriteMetadataSetArgs(bench);
+}
+
+void WriteMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+
+ int64_t file_size = 0;
+ for (auto _ : state) {
+ auto sink = benchmark.WriteFile();
+ file_size = sink->Finish().ValueOrDie()->size();
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(file_size);
+}
+
+void ReadMetadata(benchmark::State& state) {
+ MetadataBenchmark benchmark(&state);
+ auto contents = benchmark.WriteFile()->Finish().ValueOrDie();
+
+ for (auto _ : state) {
+ benchmark.ReadFile(contents);
+ }
+ state.SetItemsProcessed(state.iterations());
+ state.counters["file_size"] = static_cast<double>(contents->size());
Review Comment:
Ok, I'm not sure it makes sense, as metadata and data are quite intertwined in practice (why would we not measure page header serialization?).
The reason the benchmarks are not symmetrical is that reading and writing is not symmetrical. You have to write an entire Parquet file at once (there's no practical way to avoid that), while you can read a Parquet file piecewise or partially.
I can rename the write benchmark to something else if that's less misleading...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org