You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "sahil1105 (via GitHub)" <gi...@apache.org> on 2024/08/12 19:00:46 UTC
[PR] GH-43660 [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
sahil1105 opened a new pull request, #43661:
URL: https://github.com/apache/arrow/pull/43661
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
(See https://github.com/apache/arrow/issues/43660)
### What changes are included in this PR?
To resolve this, this PR makes the following changes:
- In `ParquetFileFormat::ScanBatchesAsync`, set the `batch_size` for the `reader` to INT64_MAX. Since we already have a `SlicingGenerator` that is responsible for converting the reader's output into `batch_size` sized batches, we don't need the reader to batch its output.
- Add a `CastingGenerator` that applies any required casts to the output of the reader _before_ the output is passed to `SlicingGenerator`. The logic for the cast closely follows the logic in `MakeExecBatch` (https://github.com/apache/arrow/blob/6a2e19a852b367c72d7b12da4d104456491ed8b7/cpp/src/arrow/compute/expression.cc#L644). There's an existing note in that function which states that the readers should be the ones performing the cast (https://github.com/apache/arrow/blob/6a2e19a852b367c72d7b12da4d104456491ed8b7/cpp/src/arrow/compute/expression.cc#L672), so I believe this change gets us closer to that goal.
### Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
I would like some feedback on the best way to test this. The changes passed our internal test cases, however, it may not account for all possible use cases.
### Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "scott-routledge2 (via GitHub)" <gi...@apache.org>.
scott-routledge2 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r2380199206
##########
cpp/src/arrow/dataset/parquet_scan_benchmark.cc:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/testing/gtest_util.h"
+#include "benchmark/benchmark.h"
+
+#include "arrow/api.h"
+#include "arrow/compute/initialize.h"
+#include "arrow/dataset/dataset.h"
+#include "arrow/dataset/file_parquet.h"
+#include "arrow/dataset/scanner.h"
+#include "arrow/io/memory.h"
+#include "parquet/arrow/writer.h"
+
+namespace arrow {
+namespace dataset {
+
+using parquet::arrow::WriteTable;
+
+Result<std::shared_ptr<Buffer>> WriteStringColParquetBuffer(int64_t nrows) {
+ auto schema = arrow::schema({arrow::field("my_string_col", arrow::utf8())});
+
+ arrow::StringBuilder builder;
+ for (int64_t i = 0; i < nrows; i++) {
+ ARROW_RETURN_NOT_OK(builder.Append("row_" + std::to_string(i)));
+ }
+ std::shared_ptr<arrow::Array> arr;
+ ARROW_RETURN_NOT_OK(builder.Finish(&arr));
+ auto table = arrow::Table::Make(schema, {arr});
+
+ ARROW_ASSIGN_OR_RAISE(auto sink, arrow::io::BufferOutputStream::Create());
+ ARROW_RETURN_NOT_OK(WriteTable(*table, arrow::default_memory_pool(), sink));
+ return sink->Finish();
+}
+
+static void ParquetScanToTableCastStrings(benchmark::State& state) {
+ // GH-43660: Scan parquet data including a String column using a dataset object with
+ // LargeString in schema.
+ int64_t nrows = 100'000;
+ int64_t batch_size = 100;
+ bool use_threads = false;
+ auto format = std::make_shared<ParquetFileFormat>();
+
+ // Create a buffer with a single String column and wrap with FileFragment
+ ASSERT_OK_AND_ASSIGN(std::shared_ptr<Buffer> buffer,
+ WriteStringColParquetBuffer(nrows));
+ auto buffer_reader = std::make_shared<arrow::io::BufferReader>(buffer);
+ FileSource source(buffer_reader, buffer->size());
+ ASSERT_OK_AND_ASSIGN(auto fragment, format->MakeFragment(source));
+ std::vector<std::shared_ptr<FileFragment>> fragments{fragment};
+
+ // Create a dataset from FileFragment and set schema to LargeString (require casting).
+ auto schema = arrow::schema({field("my_string_col", arrow::large_utf8())});
+ ASSERT_OK_AND_ASSIGN(auto dataset, FileSystemDataset::Make(
+ schema, compute::literal(true), format,
+ /*filesystem=*/nullptr, std::move(fragments)));
+
+ ASSERT_OK_AND_ASSIGN(auto builder, dataset->NewScan());
+ ASSERT_OK(builder->BatchSize(batch_size));
+ ASSERT_OK(builder->UseThreads(use_threads));
+ ASSERT_OK_AND_ASSIGN(auto scanner, builder->Finish());
+
+ for (auto _ : state) {
+ ASSERT_OK_AND_ASSIGN(auto table, scanner->ToTable());
+ benchmark::DoNotOptimize(table);
+ }
+
+ state.SetItemsProcessed(state.iterations() * nrows);
+}
+
+BENCHMARK(ParquetScanToTableCastStrings);
+
+} // namespace dataset
Review Comment:
Local results (comparing to main):
```
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (1)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
benchmark baseline contender change % counters
ParquetScanToTableCastStrings 732.326K items/sec 1.075M items/sec 46.841 {'family_index': 0, 'per_family_instance_index': 0, 'run_name': 'ParquetScanToTableCastStrings', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 5}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "scott-routledge2 (via GitHub)" <gi...@apache.org>.
scott-routledge2 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-3550256332
@pitrou @zanmato1984 Thanks for the explanations! I opened a new PR to address the casting issue as discussed. As for 2, I'm also happy to keep this PR open and investigate a cleaner way of ensuring output batches have the same schema. Although I was wondering if there should be a separate issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1715474472
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
https://github.com/apache/arrow/blob/main/cpp/src/parquet/arrow/reader_internal.cc#L474
Refer to here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1736873205
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -617,6 +684,9 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
[this, options, parquet_fragment, pre_filtered,
row_groups](const std::shared_ptr<parquet::arrow::FileReader>& reader) mutable
-> Result<RecordBatchGenerator> {
+ // Since we already do the batching through the SlicingGenerator, we don't need the
+ // reader to batch its output.
+ reader->set_batch_size(std::numeric_limits<int64_t>::max());
// Ensure that parquet_fragment has FileMetaData
Review Comment:
I think we have that problem regardless of the batch size of the reader.
We pass this reader to `reader->GetRecordBatchGenerator` (a few lines down). This eventually creates a `RowGroupGenerator` (https://github.com/apache/arrow/blob/6a2e19a852b367c72d7b12da4d104456491ed8b7/cpp/src/parquet/arrow/reader.cc#L1204). If we look at the implementation of `FetchNext` for `RowGroupGenerator`, it essentially calls `ReadOneRowGroup` which reads the entire row group into a table and then creates an output stream using `TableBatchReader` (https://github.com/apache/arrow/blob/6a2e19a852b367c72d7b12da4d104456491ed8b7/cpp/src/parquet/arrow/reader.cc#L1170). So, we're already reading an entire row group into a single table. The `batch_size` just creates a reader on top of it which generates zero-copy slices. This is the same as what the SlicingGenerator does and is redundant. In my opinion, it's better to set the batch size to INT_MAX so that it returns one table per row group, and then we can perform the batching through the SlicingGenerator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-3512163522
Ok, I think this is approaching the problem from the wrong end.
1. We should fix the issue in `CastBinaryToBinaryOffsets` so that the O(N^2) behavior cannot occur anymore
2. Also, we are now able to read Parquet [directly as LargeString](https://github.com/apache/arrow/pull/46532), so the intermediate cast is not required anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "scott-routledge2 (via GitHub)" <gi...@apache.org>.
scott-routledge2 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r2492033604
##########
cpp/src/arrow/dataset/parquet_scan_benchmark.cc:
##########
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/testing/gtest_util.h"
+#include "benchmark/benchmark.h"
+
+#include "arrow/api.h"
+#include "arrow/compute/initialize.h"
+#include "arrow/dataset/dataset.h"
+#include "arrow/dataset/file_parquet.h"
+#include "arrow/dataset/scanner.h"
+#include "arrow/io/memory.h"
+#include "parquet/arrow/writer.h"
+
+namespace arrow {
+namespace dataset {
+
+using parquet::arrow::WriteTable;
+
+Result<std::shared_ptr<Buffer>> WriteStringColParquetBuffer(int64_t nrows) {
+ auto schema = arrow::schema({arrow::field("my_string_col", arrow::utf8())});
+
+ arrow::StringBuilder builder;
+ for (int64_t i = 0; i < nrows; i++) {
+ ARROW_RETURN_NOT_OK(builder.Append("row_" + std::to_string(i)));
+ }
+ std::shared_ptr<arrow::Array> arr;
+ ARROW_RETURN_NOT_OK(builder.Finish(&arr));
+ auto table = arrow::Table::Make(schema, {arr});
+
+ ARROW_ASSIGN_OR_RAISE(auto sink, arrow::io::BufferOutputStream::Create());
+ ARROW_RETURN_NOT_OK(WriteTable(*table, arrow::default_memory_pool(), sink));
+ return sink->Finish();
+}
+
+static void ParquetScanToTableCastStrings(benchmark::State& state) {
+ // GH-43660: Scan parquet data including a String column using a dataset object with
+ // LargeString in schema.
+ size_t num_batches = state.range(0);
+ size_t batch_size = state.range(1);
+ size_t nrows = num_batches * batch_size;
+ auto format = std::make_shared<ParquetFileFormat>();
+
+ // Create a buffer with a single String column and wrap with FileFragment
+ ASSERT_OK_AND_ASSIGN(std::shared_ptr<Buffer> buffer,
+ WriteStringColParquetBuffer(nrows));
+ auto buffer_reader = std::make_shared<arrow::io::BufferReader>(buffer);
+ FileSource source(buffer_reader, buffer->size());
+ ASSERT_OK_AND_ASSIGN(auto fragment, format->MakeFragment(source));
+ std::vector<std::shared_ptr<FileFragment>> fragments{fragment};
+
+ // Create a dataset from FileFragment and set schema to LargeString (require casting).
+ auto schema = arrow::schema({field("my_string_col", arrow::large_utf8())});
+ ASSERT_OK_AND_ASSIGN(auto dataset, FileSystemDataset::Make(
+ schema, compute::literal(true), format,
+ /*filesystem=*/nullptr, std::move(fragments)));
+
+ ASSERT_OK_AND_ASSIGN(auto builder, dataset->NewScan());
+ ASSERT_OK(builder->BatchSize(batch_size));
+ ASSERT_OK_AND_ASSIGN(auto scanner, builder->Finish());
+
+ for (auto _ : state) {
+ ASSERT_OK_AND_ASSIGN(auto table, scanner->ToTable());
+ benchmark::DoNotOptimize(table);
+ }
+
+ state.SetItemsProcessed(state.iterations() * nrows);
+}
+
+static void ParquetScanBenchmark_Customize(benchmark::internal::Benchmark* b) {
+ for (const int32_t num_batches : {1000}) {
+ for (const int batch_size : {10, 100, 1000}) {
Review Comment:
Thanks for the suggestion, I updated the benchmark with those batch sizes:
```
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (3)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
benchmark baseline contender change % counters
ParquetScanToTableCastStrings/num_batches:1000/batch_size:1000 186.666K items/sec 11.289M items/sec 5947.814 {'family_index': 0, 'per_family_instance_index': 0, 'run_name': 'ParquetScanToTableCastStrings/num_batches:1000/batch_size:1000', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 1}
ParquetScanToTableCastStrings/num_batches:10/batch_size:100000 403.236M items/sec 797.722M items/sec 97.830 {'family_index': 0, 'per_family_instance_index': 2, 'run_name': 'ParquetScanToTableCastStrings/num_batches:10/batch_size:100000', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 100}
ParquetScanToTableCastStrings/num_batches:100/batch_size:10000 57.155M items/sec 110.185M items/sec 92.784 {'family_index': 0, 'per_family_instance_index': 1, 'run_name': 'ParquetScanToTableCastStrings/num_batches:100/batch_size:10000', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 39}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "scott-routledge2 (via GitHub)" <gi...@apache.org>.
scott-routledge2 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r2380199206
##########
cpp/src/arrow/dataset/parquet_scan_benchmark.cc:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/testing/gtest_util.h"
+#include "benchmark/benchmark.h"
+
+#include "arrow/api.h"
+#include "arrow/compute/initialize.h"
+#include "arrow/dataset/dataset.h"
+#include "arrow/dataset/file_parquet.h"
+#include "arrow/dataset/scanner.h"
+#include "arrow/io/memory.h"
+#include "parquet/arrow/writer.h"
+
+namespace arrow {
+namespace dataset {
+
+using parquet::arrow::WriteTable;
+
+Result<std::shared_ptr<Buffer>> WriteStringColParquetBuffer(int64_t nrows) {
+ auto schema = arrow::schema({arrow::field("my_string_col", arrow::utf8())});
+
+ arrow::StringBuilder builder;
+ for (int64_t i = 0; i < nrows; i++) {
+ ARROW_RETURN_NOT_OK(builder.Append("row_" + std::to_string(i)));
+ }
+ std::shared_ptr<arrow::Array> arr;
+ ARROW_RETURN_NOT_OK(builder.Finish(&arr));
+ auto table = arrow::Table::Make(schema, {arr});
+
+ ARROW_ASSIGN_OR_RAISE(auto sink, arrow::io::BufferOutputStream::Create());
+ ARROW_RETURN_NOT_OK(WriteTable(*table, arrow::default_memory_pool(), sink));
+ return sink->Finish();
+}
+
+static void ParquetScanToTableCastStrings(benchmark::State& state) {
+ // GH-43660: Scan parquet data including a String column using a dataset object with
+ // LargeString in schema.
+ int64_t nrows = 100'000;
+ int64_t batch_size = 100;
+ bool use_threads = false;
+ auto format = std::make_shared<ParquetFileFormat>();
+
+ // Create a buffer with a single String column and wrap with FileFragment
+ ASSERT_OK_AND_ASSIGN(std::shared_ptr<Buffer> buffer,
+ WriteStringColParquetBuffer(nrows));
+ auto buffer_reader = std::make_shared<arrow::io::BufferReader>(buffer);
+ FileSource source(buffer_reader, buffer->size());
+ ASSERT_OK_AND_ASSIGN(auto fragment, format->MakeFragment(source));
+ std::vector<std::shared_ptr<FileFragment>> fragments{fragment};
+
+ // Create a dataset from FileFragment and set schema to LargeString (require casting).
+ auto schema = arrow::schema({field("my_string_col", arrow::large_utf8())});
+ ASSERT_OK_AND_ASSIGN(auto dataset, FileSystemDataset::Make(
+ schema, compute::literal(true), format,
+ /*filesystem=*/nullptr, std::move(fragments)));
+
+ ASSERT_OK_AND_ASSIGN(auto builder, dataset->NewScan());
+ ASSERT_OK(builder->BatchSize(batch_size));
+ ASSERT_OK(builder->UseThreads(use_threads));
+ ASSERT_OK_AND_ASSIGN(auto scanner, builder->Finish());
+
+ for (auto _ : state) {
+ ASSERT_OK_AND_ASSIGN(auto table, scanner->ToTable());
+ benchmark::DoNotOptimize(table);
+ }
+
+ state.SetItemsProcessed(state.iterations() * nrows);
+}
+
+BENCHMARK(ParquetScanToTableCastStrings);
+
+} // namespace dataset
Review Comment:
Local results (comparing to main):
```
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Non-regressions: (20)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
benchmark baseline contender change % counters
ParquetScanToTableCastStrings/num_batches:1000/batch_size:1000 1.068M items/sec 10.574M items/sec 890.391 {'family_index': 0, 'per_family_instance_index': 2, 'run_name': 'ParquetScanToTableCastStrings/num_batches:1000/batch_size:1000', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 1}
ParquetScanToTableCastStrings/num_batches:1000/batch_size:100 761.915K items/sec 1.065M items/sec 39.760 {'family_index': 0, 'per_family_instance_index': 1, 'run_name': 'ParquetScanToTableCastStrings/num_batches:1000/batch_size:100', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 4}
ParquetScanToTableCastStrings/num_batches:1000/batch_size:10 86.631K items/sec 107.530K items/sec 24.124 {'family_index': 0, 'per_family_instance_index': 0, 'run_name': 'ParquetScanToTableCastStrings/num_batches:1000/batch_size:10', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 5}
ScanOnlyBench/num_batches:1000/batch_size:1000/scan_alg:0/real_time 41.041 MiB/sec 49.586 MiB/sec 20.820 {'family_index': 1, 'per_family_instance_index': 4, 'run_name': 'ScanOnlyBench/num_batches:1000/batch_size:1000/scan_alg:0/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 6}
ScanOnlyBench/num_batches:1000/batch_size:10/scan_alg:0/real_time 457.346 KiB/sec 518.384 KiB/sec 13.346 {'family_index': 1, 'per_family_instance_index': 0, 'run_name': 'ScanOnlyBench/num_batches:1000/batch_size:10/scan_alg:0/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 6}
ScanOnlyBench/num_batches:1000/batch_size:100/scan_alg:0/real_time 4.421 MiB/sec 5.011 MiB/sec 13.339 {'family_index': 1, 'per_family_instance_index': 2, 'run_name': 'ScanOnlyBench/num_batches:1000/batch_size:100/scan_alg:0/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 6}
ScanOnlyBench/num_batches:1000/batch_size:10/scan_alg:1/real_time 634.336 KiB/sec 718.653 KiB/sec 13.292 {'family_index': 1, 'per_family_instance_index': 1, 'run_name': 'ScanOnlyBench/num_batches:1000/batch_size:10/scan_alg:1/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 9}
MinimalEndToEndBench/num_batches:1000/batch_size:100/scan_alg:0/real_time 4.563 MiB/sec 5.089 MiB/sec 11.535 {'family_index': 0, 'per_family_instance_index': 2, 'run_name': 'MinimalEndToEndBench/num_batches:1000/batch_size:100/scan_alg:0/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 7}
MinimalEndToEndBench/num_batches:1000/batch_size:10/scan_alg:1/real_time 624.277 KiB/sec 680.752 KiB/sec 9.047 {'family_index': 0, 'per_family_instance_index': 1, 'run_name': 'MinimalEndToEndBench/num_batches:1000/batch_size:10/scan_alg:1/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 9}
MinimalEndToEndBench/num_batches:1000/batch_size:10/scan_alg:0/real_time 483.179 KiB/sec 522.102 KiB/sec 8.055 {'family_index': 0, 'per_family_instance_index': 0, 'run_name': 'MinimalEndToEndBench/num_batches:1000/batch_size:10/scan_alg:0/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 7}
ScanOnlyBench/num_batches:1000/batch_size:100/scan_alg:1/real_time 6.578 MiB/sec 7.080 MiB/sec 7.645 {'family_index': 1, 'per_family_instance_index': 3, 'run_name': 'ScanOnlyBench/num_batches:1000/batch_size:100/scan_alg:1/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 9}
MinimalEndToEndBench/num_batches:1000/batch_size:1000/scan_alg:0/real_time 46.819 MiB/sec 49.794 MiB/sec 6.353 {'family_index': 0, 'per_family_instance_index': 4, 'run_name': 'MinimalEndToEndBench/num_batches:1000/batch_size:1000/scan_alg:0/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 7}
ScanOnlyBench/num_batches:1000/batch_size:1000/scan_alg:1/real_time 67.173 MiB/sec 70.629 MiB/sec 5.145 {'family_index': 1, 'per_family_instance_index': 5, 'run_name': 'ScanOnlyBench/num_batches:1000/batch_size:1000/scan_alg:1/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 9}
GetFilteredFragments/single_file 1.501M items/sec 1.557M items/sec 3.709 {'family_index': 3, 'per_family_instance_index': 0, 'run_name': 'GetFilteredFragments/single_file', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 105, 'num_filtered_fragments': 1.0, 'num_fragments': 10000.0}
MinimalEndToEndBench/num_batches:1000/batch_size:100/scan_alg:1/real_time 6.327 MiB/sec 6.481 MiB/sec 2.430 {'family_index': 0, 'per_family_instance_index': 3, 'run_name': 'MinimalEndToEndBench/num_batches:1000/batch_size:100/scan_alg:1/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 9}
GetFilteredFragments/range 229.364K items/sec 232.693K items/sec 1.451 {'family_index': 4, 'per_family_instance_index': 0, 'run_name': 'GetFilteredFragments/range', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 16, 'num_filtered_fragments': 9800.0, 'num_fragments': 10000.0}
MinimalEndToEndBench/num_batches:1000/batch_size:1000/scan_alg:1/real_time 61.721 MiB/sec 62.435 MiB/sec 1.156 {'family_index': 0, 'per_family_instance_index': 5, 'run_name': 'MinimalEndToEndBench/num_batches:1000/batch_size:1000/scan_alg:1/real_time', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 9}
GetFilteredFragments/single_dir 3.419M items/sec 3.444M items/sec 0.733 {'family_index': 1, 'per_family_instance_index': 0, 'run_name': 'GetFilteredFragments/single_dir', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 236, 'num_filtered_fragments': 100.0, 'num_fragments': 10000.0}
GetFilteredFragments/multi_dir 40.060K items/sec 40.215K items/sec 0.386 {'family_index': 2, 'per_family_instance_index': 0, 'run_name': 'GetFilteredFragments/multi_dir', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 3, 'num_filtered_fragments': 100.0, 'num_fragments': 10000.0}
GetAllFragments 35.682M items/sec 35.433M items/sec -0.698 {'family_index': 0, 'per_family_instance_index': 0, 'run_name': 'GetAllFragments', 'repetitions': 1, 'repetition_index': 0, 'threads': 1, 'iterations': 2282, 'num_fragments': 10000.0}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1715585790
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
> Based on what I see, that is only responsible for casting the data to the logical type specified in the parquet metadata and not the Arrow type we want to convert to (the one in the dataset_schema)
Parquet logical type doesn't have an arrow schema, isn't it? Binary reader reads from `::arrow::BinaryBuilder`, and casting it to user-specified binary type.
> For strings, that seems to always map to a String type (based on FromByteArray which is called by GetArrowType which is called by GetTypeForNode which is called by NodeToSchemaField which is called in SchemaManifest::Make during the creation of the LeafReader).
Yeah, you're right, the read "cast" with file-schema rather than an expected schema. I think a native cast is better here but this doesn't solve your problem, perhaps I can trying to add a naive `SchemaManifest` with hint solving here, but it would spend some time.
```
::arrow::Result<std::shared_ptr<ArrowType>> GetTypeForNode(
int column_index, const schema::PrimitiveNode& primitive_node,
SchemaTreeContext* ctx)
```
Maybe we should rethink the `GetTypeForNode` handling for string/large_string/stringView, or using some handle written type hint here. A casting generator is also good for me when the reader cannot provide the right casting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1736860944
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,66 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
+ CastingGenerator(RecordBatchGenerator source, std::shared_ptr<Schema> final_schema,
+ const std::unordered_set<std::string>& cols_to_skip,
+ MemoryPool* pool = default_memory_pool())
+ : source_(source),
+ final_schema_(final_schema),
+ cols_to_skip_(cols_to_skip),
+ exec_ctx(std::make_shared<compute::ExecContext>(pool)) {}
+
+ Future<std::shared_ptr<RecordBatch>> operator()() {
+ return this->source_().Then([this](const std::shared_ptr<RecordBatch>& next)
+ -> Result<std::shared_ptr<RecordBatch>> {
+ if (IsIterationEnd(next) || this->final_schema_ == nullptr) {
+ return next;
+ }
+ std::vector<std::shared_ptr<Array>> out_cols;
+ std::vector<std::shared_ptr<Field>> out_schema_fields;
+
+ bool changed = false;
+ for (const auto& field : this->final_schema_->fields()) {
+ FieldRef field_ref = FieldRef(field->name());
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<Array> column,
+ field_ref.GetOneOrNone(*next));
+ if (column) {
+ if (this->cols_to_skip_.count(field->name())) {
+ // Maintain the original input type.
+ out_schema_fields.emplace_back(field->WithType(column->type()));
+ out_cols.emplace_back(std::move(column));
Review Comment:
Yes, to skip the cast and leave them as they are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1736878509
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -637,8 +707,17 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
reader->GetRecordBatchGenerator(
reader, row_groups, column_projection,
::arrow::internal::GetCpuThreadPool(), rows_to_readahead));
+ // We need to skip casting the dictionary columns since the dataset_schema doesn't
+ // have the dictionary-encoding information. Parquet reader will return them with the
+ // dictionary type, which is what we eventually want.
Review Comment:
I wasn't sure, so I left that case untouched. There are more casts done further up the chain (e.g. in `MakeExecBatch` potentially) that seem to handle those cases. I couldn't figure out how the dictionary case gets handled either, so I left it as is. I'm happy to implement it here if you could point me in the right direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "scott-routledge2 (via GitHub)" <gi...@apache.org>.
scott-routledge2 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-3329056349
cc @bkietz , @pitrou
Hello! I was interested in picking up this PR from Sahil since the use case still seems relevant to us. I tested locally with the latest changes on main and it looked like this PR can improve performance of the Scanner quite a bit, although there were some concerns about the robustness of setting the batch size to INT_MAX in `ScanBatchesAsync`.
Was wondering if I could get some feedback on whether this is headed in the right direction. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "scott-routledge2 (via GitHub)" <gi...@apache.org>.
scott-routledge2 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-3518523113
> Ok, I think this is approaching the problem from the wrong end.
>
> 1. We should fix the issue in `CastBinaryToBinaryOffsets` so that the O(N^2) behavior cannot occur anymore
>
> 2. Also, we are now able to read Parquet [directly as LargeString](https://github.com/apache/arrow/pull/46532), so the intermediate cast is not required anymore
I would be happy to open a separate issue to look at `CastBinaryToBinaryOffsets`, however, I wasn't sure how to make this cast more efficient without changing the API significantly? A single cast of a slice is O(offset + length), since a new buffer for the string offsets needs to be created with the same shape as the original slice. Casting slices of a size N array only becomes O(N^2) when length of the slices << N, so to me, the most straightforward approach would be to change how the casting is being invoked i.e. casting the entire array at once vs slices.
For the second point, the cast here was not a workaround for reading LargeString data, but rather, we were working with datasets that have a single, unified schema and individual files that potentially have different schemas (e.g. string vs large_string ), so the casting would still be necessary at some point to reconcile the differences in schema. This cast is currently handled in [MakeExecBatch](https://github.com/apache/arrow/blob/6a2e19a852b367c72d7b12da4d104456491ed8b7/cpp/src/arrow/compute/expression.cc#L672), but there is comment that this should be done by the reader, so I believe this PR would be an incremental step in achieving this goal, but I was wondering what your thoughts would be?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r2451510870
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -627,6 +693,9 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
[this, options, parquet_fragment, pre_filtered,
row_groups](const std::shared_ptr<parquet::arrow::FileReader>& reader) mutable
-> Result<RecordBatchGenerator> {
+ // Since we already do the batching through the SlicingGenerator, we don't need the
+ // reader to batch its output.
+ reader->set_batch_size(std::numeric_limits<int64_t>::max());
Review Comment:
Well, if you have a very large row group (say 1 GB or more), what this will do is read the entire file at once, then slice it, instead of generating batches progressively.
So this sounds like a bad idea to me. The entire purpose of `ScanBatchesAsync` is to generate batches progressively so that the consumer can process them without waiting for the entire file to be read. This allows covering IO latencies and also decreasing memory footprint (say you're computing the average of a column: it would be wasteful to load the entire column at once instead of having a running average computation over smaller batches).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1714616471
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
I'm a bit curious why casting generator is required, since during reading parquet to arrow, Parquet reader already applies a round of casting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-2286895461
I'm not sure why the "Java JNI / AMD64 manylinux2014 Java JNI (pull_request)" and "continuous-integration/appveyor/pr" are failing. The latter in particular passed on the previous commit which is the same as this commit (my last commit was an empty commit to trigger the CI again). Any insights would be appreciated.
I would also appreciate feedback on if and what unit tests need to be added for this change. It seems like the existing unit tests provide good coverage of this code (they helped me find some of the bugs), but I'm happy to add more if it is deemed useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1715623370
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
Provide hint would looks like: https://github.com/apache/arrow-rs/pull/5939
Maybe I can add separate issue for that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-2308001919
Sorry for delaying I'll take a pass today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r2451523468
##########
cpp/src/arrow/dataset/parquet_scan_benchmark.cc:
##########
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/testing/gtest_util.h"
+#include "benchmark/benchmark.h"
+
+#include "arrow/api.h"
+#include "arrow/compute/initialize.h"
+#include "arrow/dataset/dataset.h"
+#include "arrow/dataset/file_parquet.h"
+#include "arrow/dataset/scanner.h"
+#include "arrow/io/memory.h"
+#include "parquet/arrow/writer.h"
+
+namespace arrow {
+namespace dataset {
+
+using parquet::arrow::WriteTable;
+
+Result<std::shared_ptr<Buffer>> WriteStringColParquetBuffer(int64_t nrows) {
+ auto schema = arrow::schema({arrow::field("my_string_col", arrow::utf8())});
+
+ arrow::StringBuilder builder;
+ for (int64_t i = 0; i < nrows; i++) {
+ ARROW_RETURN_NOT_OK(builder.Append("row_" + std::to_string(i)));
+ }
+ std::shared_ptr<arrow::Array> arr;
+ ARROW_RETURN_NOT_OK(builder.Finish(&arr));
+ auto table = arrow::Table::Make(schema, {arr});
+
+ ARROW_ASSIGN_OR_RAISE(auto sink, arrow::io::BufferOutputStream::Create());
+ ARROW_RETURN_NOT_OK(WriteTable(*table, arrow::default_memory_pool(), sink));
+ return sink->Finish();
+}
+
+static void ParquetScanToTableCastStrings(benchmark::State& state) {
+ // GH-43660: Scan parquet data including a String column using a dataset object with
+ // LargeString in schema.
+ size_t num_batches = state.range(0);
+ size_t batch_size = state.range(1);
+ size_t nrows = num_batches * batch_size;
+ auto format = std::make_shared<ParquetFileFormat>();
+
+ // Create a buffer with a single String column and wrap with FileFragment
+ ASSERT_OK_AND_ASSIGN(std::shared_ptr<Buffer> buffer,
+ WriteStringColParquetBuffer(nrows));
+ auto buffer_reader = std::make_shared<arrow::io::BufferReader>(buffer);
+ FileSource source(buffer_reader, buffer->size());
+ ASSERT_OK_AND_ASSIGN(auto fragment, format->MakeFragment(source));
+ std::vector<std::shared_ptr<FileFragment>> fragments{fragment};
+
+ // Create a dataset from FileFragment and set schema to LargeString (require casting).
+ auto schema = arrow::schema({field("my_string_col", arrow::large_utf8())});
+ ASSERT_OK_AND_ASSIGN(auto dataset, FileSystemDataset::Make(
+ schema, compute::literal(true), format,
+ /*filesystem=*/nullptr, std::move(fragments)));
+
+ ASSERT_OK_AND_ASSIGN(auto builder, dataset->NewScan());
+ ASSERT_OK(builder->BatchSize(batch_size));
+ ASSERT_OK_AND_ASSIGN(auto scanner, builder->Finish());
+
+ for (auto _ : state) {
+ ASSERT_OK_AND_ASSIGN(auto table, scanner->ToTable());
+ benchmark::DoNotOptimize(table);
+ }
+
+ state.SetItemsProcessed(state.iterations() * nrows);
+}
+
+static void ParquetScanBenchmark_Customize(benchmark::internal::Benchmark* b) {
+ for (const int32_t num_batches : {1000}) {
+ for (const int batch_size : {10, 100, 1000}) {
Review Comment:
Those batch sizes are much too small for good performance, so I'm not sure why they're being benchmarked.
You could instead try `{1000, 10000, 100000}`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r2510843507
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -627,6 +693,9 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
[this, options, parquet_fragment, pre_filtered,
row_groups](const std::shared_ptr<parquet::arrow::FileReader>& reader) mutable
-> Result<RecordBatchGenerator> {
+ // Since we already do the batching through the SlicingGenerator, we don't need the
+ // reader to batch its output.
+ reader->set_batch_size(std::numeric_limits<int64_t>::max());
Review Comment:
Thanks for the explanation @scott-routledge2 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1715557796
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
Thank you!
Based on what I see, that is only responsible for casting the data to the logical type specified in the parquet metadata and not the Arrow type we want to convert to (the one in the dataset_schema). For strings, that seems to always map to a String type (based on `FromByteArray` which is called by `GetArrowType` which is called by `GetTypeForNode` which is called by `NodeToSchemaField` which is called in `SchemaManifest::Make` during the creation of the `LeafReader`). Am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1718326215
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
Sounds good, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1730212079
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,66 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
+ CastingGenerator(RecordBatchGenerator source, std::shared_ptr<Schema> final_schema,
+ const std::unordered_set<std::string>& cols_to_skip,
+ MemoryPool* pool = default_memory_pool())
+ : source_(source),
+ final_schema_(final_schema),
Review Comment:
```suggestion
final_schema_(std::move(final_schema)),
```
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -617,6 +684,9 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
[this, options, parquet_fragment, pre_filtered,
row_groups](const std::shared_ptr<parquet::arrow::FileReader>& reader) mutable
-> Result<RecordBatchGenerator> {
+ // Since we already do the batching through the SlicingGenerator, we don't need the
+ // reader to batch its output.
+ reader->set_batch_size(std::numeric_limits<int64_t>::max());
// Ensure that parquet_fragment has FileMetaData
Review Comment:
Assuming a large file, would memory usage grows high in this case?
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,66 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
+ CastingGenerator(RecordBatchGenerator source, std::shared_ptr<Schema> final_schema,
+ const std::unordered_set<std::string>& cols_to_skip,
+ MemoryPool* pool = default_memory_pool())
+ : source_(source),
+ final_schema_(final_schema),
+ cols_to_skip_(cols_to_skip),
+ exec_ctx(std::make_shared<compute::ExecContext>(pool)) {}
+
+ Future<std::shared_ptr<RecordBatch>> operator()() {
+ return this->source_().Then([this](const std::shared_ptr<RecordBatch>& next)
+ -> Result<std::shared_ptr<RecordBatch>> {
+ if (IsIterationEnd(next) || this->final_schema_ == nullptr) {
+ return next;
+ }
+ std::vector<std::shared_ptr<Array>> out_cols;
+ std::vector<std::shared_ptr<Field>> out_schema_fields;
+
+ bool changed = false;
+ for (const auto& field : this->final_schema_->fields()) {
+ FieldRef field_ref = FieldRef(field->name());
+ ARROW_ASSIGN_OR_RAISE(std::shared_ptr<Array> column,
+ field_ref.GetOneOrNone(*next));
+ if (column) {
+ if (this->cols_to_skip_.count(field->name())) {
+ // Maintain the original input type.
+ out_schema_fields.emplace_back(field->WithType(column->type()));
+ out_cols.emplace_back(std::move(column));
Review Comment:
So `cols_to_skip_` is just not "Project" this field, rather than not need this field?
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -637,8 +707,17 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
reader->GetRecordBatchGenerator(
reader, row_groups, column_projection,
::arrow::internal::GetCpuThreadPool(), rows_to_readahead));
+ // We need to skip casting the dictionary columns since the dataset_schema doesn't
+ // have the dictionary-encoding information. Parquet reader will return them with the
+ // dictionary type, which is what we eventually want.
Review Comment:
I've forget something here, would `Dict(String)` and `Cast(xxx -> LargeString)` matters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-2307672281
@mapleFU If you get a chance to review this PR this weekend, that'd be great. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660 [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-2284713064
:warning: GitHub issue #43660 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "srilman (via GitHub)" <gi...@apache.org>.
srilman commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-2842188178
@scott-routledge2 @IsaacWarren
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-3550304649
> Although I was wondering if there should be a separate issue?
I think so. Moving on to a separate issue allow us to preserve the original purpose along with all the discussion history in the current issue/pr, which would make sense to the future readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r2451530134
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -649,8 +718,17 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
ARROW_ASSIGN_OR_RAISE(auto generator, reader->GetRecordBatchGenerator(
reader, row_groups, column_projection,
cpu_executor, rows_to_readahead));
+ // We need to skip casting the dictionary columns since the dataset_schema doesn't
+ // have the dictionary-encoding information. Parquet reader will return them with the
+ // dictionary type, which is what we eventually want.
+ const std::unordered_set<std::string>& dict_cols =
+ parquet_fragment->parquet_format_.reader_options.dict_columns;
+ // Casting before slicing is more efficient. Casts on slices might require wasteful
+ // allocations and computation.
Review Comment:
Can you explain which kind of wasteful allocations and computations would be required when casting a slice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1715317034
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
You're probably right and it likely does perform some casts during the read. However, it doesn't seem to be doing it for certain cases like String to LargeString.
Could you point me to where the Parquet Reader should be performing this cast? I'm happy to add it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-2290708741
The interface general LGTM. I'm a little busy on working days, and will take a careful around in weekend. You also can mark it as ready for review here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "scott-routledge2 (via GitHub)" <gi...@apache.org>.
scott-routledge2 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r2466080647
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -627,6 +693,9 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
[this, options, parquet_fragment, pre_filtered,
row_groups](const std::shared_ptr<parquet::arrow::FileReader>& reader) mutable
-> Result<RecordBatchGenerator> {
+ // Since we already do the batching through the SlicingGenerator, we don't need the
+ // reader to batch its output.
+ reader->set_batch_size(std::numeric_limits<int64_t>::max());
Review Comment:
Based on my understanding of this example, wouldn't calling `ScanBatchesAsync` create a [`RowGroupGenerator`](https://github.com/apache/arrow/blob/6a2e19a852b367c72d7b12da4d104456491ed8b7/cpp/src/parquet/arrow/reader.cc#L1204), which would then call [`ReadOneRowGroup`](https://github.com/apache/arrow/blob/6a2e19a852b367c72d7b12da4d104456491ed8b7/cpp/src/parquet/arrow/reader.cc#L1170) and read the entire row group into an in-memory table anyways? The batch size here seems to just control the chunk size of that table.
The goal of this PR was to reduce the overhead of casting batches of string offset buffers (which would require O(N^2) memory/compute where N is the length of a row group) by performing the cast on the entire table/row-group at once, which we will have at some point because of `ReadOneRowGroup`, so I don't think this represents a regression. However, I understand wanting to keep the intended semantics of `ScanBatchesAsync` the same. Maybe instead we can perform the cast at a different layer of the stack e.g. at the reader level?
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -627,6 +693,9 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
[this, options, parquet_fragment, pre_filtered,
row_groups](const std::shared_ptr<parquet::arrow::FileReader>& reader) mutable
-> Result<RecordBatchGenerator> {
+ // Since we already do the batching through the SlicingGenerator, we don't need the
+ // reader to batch its output.
+ reader->set_batch_size(std::numeric_limits<int64_t>::max());
Review Comment:
Based on my understanding of this example, wouldn't calling `ScanBatchesAsync` create a [`RowGroupGenerator`](https://github.com/apache/arrow/blob/6a2e19a852b367c72d7b12da4d104456491ed8b7/cpp/src/parquet/arrow/reader.cc#L1204), which would then call [`ReadOneRowGroup`](https://github.com/apache/arrow/blob/6a2e19a852b367c72d7b12da4d104456491ed8b7/cpp/src/parquet/arrow/reader.cc#L1170) and read the entire row group into an in-memory table anyways? The batch size here seems to just control the chunk size of that table.
The goal of this PR was to reduce the overhead of casting batches of string offset buffers (which would require O(N^2) memory/compute where N is the length of a row group) by performing the cast on the entire table/row-group at once, which we will have at some point because of `ReadOneRowGroup`, so I don't think this represents a regression.
However, I understand wanting to keep the intended semantics of `ScanBatchesAsync` the same. Maybe instead we can perform the cast at a different layer of the stack e.g. at the reader level?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-3549830139
Thanks for the explanation @scott-routledge2 ! I now see the problem and sorry for you to have to illustrate that again.
IIUC there are two aspects in this particular use case:
1. The cast implementation of `utf8` to `large_utf8` for a slice with non-zero offset exhibits a performance penalty of over-allocation and redundant zero-filling.
2. The cast is implicitly relying on the thin `MakeExecBatch` doing a reluctant schema aligning.
I think they are separated. And for 1, I agree with @pitrou that it is an arguable implementation choice we made for `BinaryToBinaryCastExec`. In other words, quote:
> since the output of the cast inherits the offset from the input slice
It doesn't necessarily have to be the case. We are well free to output an intact, zero-offset array, which is logically equal to the current output with non-zero offset. This should require no API changes.
For 2, I think the requirement of outputting batches of unified schema (applying implicit casts when necessary) makes a lot of sense. However I'm wondering if we can doing it in an less-intrusive way, for example, applying the casts explicitly by leveraging existing mechanisms like `ScanOptions::projection` (https://github.com/apache/arrow/blob/5a480444da35fa26bc6952755510ad39df9f7002/cpp/src/arrow/dataset/scanner.h#L62).
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 closed pull request #43661: GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing
URL: https://github.com/apache/arrow/pull/43661
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1715751879
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
> Parquet logical type doesn't have an arrow schema, isn't it?
As far as I understand, the parquet metadata may or may not have the arrow schema. I believe it depends on the writer. It looks like it tries to get that using `GetOriginSchema` in `SchemaManifest::Make`. However, the schema at write time might not be the same as the schema the reader expects.
> Binary reader reads from ::arrow::BinaryBuilder, and casting it to user-specified binary type.
Sorry, I didn't quite follow. Are you saying that we should use this to do the cast at read time somehow?
> I think a native cast is better here but this doesn't solve your problem, perhaps I can trying to add a naive SchemaManifest with hint solving here, but it would spend some time.
> Maybe we should rethink the GetTypeForNode handling for string/large_string/stringView, or using some handle written type hint here.
That makes sense to me.
> Maybe I can add separate issue for that
That would be great, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1715585790
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
> Based on what I see, that is only responsible for casting the data to the logical type specified in the parquet metadata and not the Arrow type we want to convert to (the one in the dataset_schema)
Parquet logical type doesn't have an arrow schema, isn't it? Binary reader reads from `::arrow::BinaryBuilder`, and casting it to user-specified binary type.
> For strings, that seems to always map to a String type (based on FromByteArray which is called by GetArrowType which is called by GetTypeForNode which is called by NodeToSchemaField which is called in SchemaManifest::Make during the creation of the LeafReader).
Yeah, you're right, the read "cast" with file-schema rather than an expected schema. I think a native cast is better here but this doesn't solve your problem, perhaps I can trying to add a naive `SchemaManifest` with hint solving here, but it would spend some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-2435668571
cc @srilman
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "scott-routledge2 (via GitHub)" <gi...@apache.org>.
scott-routledge2 commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r2465960489
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -649,8 +718,17 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
ARROW_ASSIGN_OR_RAISE(auto generator, reader->GetRecordBatchGenerator(
reader, row_groups, column_projection,
cpu_executor, rows_to_readahead));
+ // We need to skip casting the dictionary columns since the dataset_schema doesn't
+ // have the dictionary-encoding information. Parquet reader will return them with the
+ // dictionary type, which is what we eventually want.
+ const std::unordered_set<std::string>& dict_cols =
+ parquet_fragment->parquet_format_.reader_options.dict_columns;
+ // Casting before slicing is more efficient. Casts on slices might require wasteful
+ // allocations and computation.
Review Comment:
When casting the string offsets buffer, we allocate `offset + length + 1` elements, memset the first `offset` elements to zero, and perform the cast on `length` many elements. So assuming batch_size = 1000, a row group of 10,000 elements would require allocating buffers of lengths 1001, 2001, ... 10,001 to perform the casts on all batches.
https://github.com/apache/arrow/blob/6a2e19a852b367c72d7b12da4d104456491ed8b7/cpp/src/arrow/compute/kernels/scalar_cast_string.cc#L251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "scott-routledge2 (via GitHub)" <gi...@apache.org>.
scott-routledge2 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-3549132175
> Could you clarify how the `O(N^2)` complexity arises? I read the issue description but couldn’t quite follow it - probably because I’m not very familiar with the dataset code - so a short walkthrough would be really helpful.
@zanmato1984 of course! It's not so much an issue with datasets as it is the actual casting kernel in compute as @pitrou pointed out.
When casting small slices of size L of a largee array of length N, to cast the first slice, we have to allocate a new offsets buffer for the up/downcasted int offsets that has size L+1. The second slice will have offset L, and [because of the way casting is implemented](https://github.com/bodo-ai/arrow-bodo-fork/blob/5eaf553bfc7aa639fd67bd622b6b808e71fbba39/cpp/src/arrow/compute/kernels/scalar_cast_string.cc#L294), we will actually need to allocate a buffer of size 2L+1 and zero out the first L elements since the output of the cast inherits the offset from the input slice. Continuing this pattern, the amount of work required is roughly, L + 2L + 3L + ... N ~ N^2/2L or O(N^2) if L is constant.
Where this relates to datasets is when reading a parquet file, ScanAsyncBatches essentially is reading one row group at a time, and then produces slices on top that row group, which then need to be casted in MakeExecBatch, following the pattern described above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-2325016641
Also cc @bkietz as the expert here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1717989511
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
I just check this. We can first add a `CastingGenerator`, because sometimes we would have schema evolution here, and need cast to a "final type". The Parquet reader code can be regard as an optimization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43661:
URL: https://github.com/apache/arrow/pull/43661#discussion_r1715585790
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
> Based on what I see, that is only responsible for casting the data to the logical type specified in the parquet metadata and not the Arrow type we want to convert to (the one in the dataset_schema)
Parquet logical type doesn't have an arrow schema, isn't it? Binary reader reads from `::arrow::BinaryBuilder`, and casting it to user-specified binary type.
> For strings, that seems to always map to a String type (based on FromByteArray which is called by GetArrowType which is called by GetTypeForNode which is called by NodeToSchemaField which is called in SchemaManifest::Make during the creation of the LeafReader).
Yeah, you're right, the read "cast" with file-schema rather than an expected schema. I think a native cast is better here but this doesn't solve your problem, perhaps I can trying to add a naive `SchemaManifest` with hint solving here, but it would spend some time.
```
::arrow::Result<std::shared_ptr<ArrowType>> GetTypeForNode(
int column_index, const schema::PrimitiveNode& primitive_node,
SchemaTreeContext* ctx)
```
Maybe we should rethink the `GetTypeForNode` handling for string/large_string/stringView, or using some handle written types
##########
cpp/src/arrow/dataset/file_parquet.cc:
##########
@@ -555,6 +562,57 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
});
}
+struct CastingGenerator {
Review Comment:
> Based on what I see, that is only responsible for casting the data to the logical type specified in the parquet metadata and not the Arrow type we want to convert to (the one in the dataset_schema)
Parquet logical type doesn't have an arrow schema, isn't it? Binary reader reads from `::arrow::BinaryBuilder`, and casting it to user-specified binary type.
> For strings, that seems to always map to a String type (based on FromByteArray which is called by GetArrowType which is called by GetTypeForNode which is called by NodeToSchemaField which is called in SchemaManifest::Make during the creation of the LeafReader).
Yeah, you're right, the read "cast" with file-schema rather than an expected schema. I think a native cast is better here but this doesn't solve your problem, perhaps I can trying to add a naive `SchemaManifest` with hint solving here, but it would spend some time.
```
::arrow::Result<std::shared_ptr<ArrowType>> GetTypeForNode(
int column_index, const schema::PrimitiveNode& primitive_node,
SchemaTreeContext* ctx)
```
Maybe we should rethink the `GetTypeForNode` handling for string/large_string/stringView, or using some handle written type hint here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "sahil1105 (via GitHub)" <gi...@apache.org>.
sahil1105 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-2291168169
> The interface general LGTM. I'm a little busy on working days, and will take a careful around in weekend. You also can mark it as ready for review here
Thanks, I appreciate it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-3664127687
Issue resolved by #48171, so closing this for now. Feel free to file a new issue/PR for the schema aligning issue, if it still is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43660: [C++] Add a `CastingGenerator` to Parquet Reader that applies required casts before slicing [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43661:
URL: https://github.com/apache/arrow/pull/43661#issuecomment-3520514776
> I would be happy to open a separate issue to look at `CastBinaryToBinaryOffsets`, however, I wasn't sure how to make this cast more efficient without changing the API significantly? A single cast of a slice is O(offset + length), since a new buffer for the string offsets needs to be created with the same shape as the original slice.
No API needs to be changed here. The output of casting is by definition the same *logical* length as the input, but it does not need to have the same physical allocation shape (i.e. offset).
The current code reuses the input offset (through `ZeroCopyCastExec`) mostly because it's easier. But it should be simple as well to produce the output with a different offset (not necessarily 0, because we would like to reuse the null bitmap and that implies we must keep `offset % 8` the same).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org