You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/20 14:40:15 UTC
[GitHub] [arrow] lidavidm opened a new pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
lidavidm opened a new pull request #11189:
URL: https://github.com/apache/arrow/pull/11189
This creates a new concatenating generator which converts AsyncGenerator<optional<vector<T>>> to AsyncGenerator<T> and supports async-reentrancy. When combined with a readahead generator, this enables us to get parallelism in scanning Parquet files.
This also properly chunks each row group into smaller batches as mentioned in ARROW-14024, though we can add more specific tests in that JIRA (and fix the issue for IPC as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #11189:
URL: https://github.com/apache/arrow/pull/11189
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-924954728
I think the segfault is the same one as in https://github.com/apache/arrow/pull/11207/commits/b3e61169d621205394d28f2b6e4b515896337e5e which I've (hopefully) fixed in #11207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-922992268
Benchmark runs are scheduled for baseline = 7128b5eb4302a768e48e0a4718b445495fc3e107 and contender = dc36f09c16a2933614b41f8912968cf1cfb364d8. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Provided benchmark filters do not have any benchmark groups to be executed on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/14909b12ff3b44afb6c2be83853db45b...deb4ec4f85ba4cab97cdc36a4c2d5d9b/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/f7dbbd8836ac43f69a52b17caa64dd26...f0e710cddcef40e89344f39f2dd0e6ba/)
[Skipped :warning: Only ['C++', 'Java'] langs are supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/2cb2668a5e37451f9d68770ce2a12a09...16fc2e6d8ec346679826ead5c66f42b7/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-933407128
Benchmark runs are scheduled for baseline = dd6c325afb6f090c9e4fb72ce6068c94857fb7bd and contender = deaab5b53957772a643fcb72fe6669e72637784d. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Provided benchmark filters do not have any benchmark groups to be executed on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/f6660218c4d64594b8b96f3d37f37165...4df79e9dddf94ed8b93f69b7e698fb2f/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/27344c56dee34c0a9604e3f4734357e9...43d9df0c39554f49b1bcc238a21fd4eb/)
[Skipped :warning: Only ['C++', 'Java'] langs are supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9f1f7109916a4b1b8f3a02bb07961581...75ad6e25a3fd460592dbb080b419749a/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-933407128
Benchmark runs are scheduled for baseline = dd6c325afb6f090c9e4fb72ce6068c94857fb7bd and contender = deaab5b53957772a643fcb72fe6669e72637784d. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Provided benchmark filters do not have any benchmark groups to be executed on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/f6660218c4d64594b8b96f3d37f37165...4df79e9dddf94ed8b93f69b7e698fb2f/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/27344c56dee34c0a9604e3f4734357e9...43d9df0c39554f49b1bcc238a21fd4eb/)
[Skipped :warning: Only ['C++', 'Java'] langs are supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9f1f7109916a4b1b8f3a02bb07961581...75ad6e25a3fd460592dbb080b419749a/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-923201576
Though for what it's worth, arrowbench gives rather similar results locally using `as.data.frame(run_benchmark(dataset_taxi_parquet, n_iter=5, cpu_count=c(4)))`.
Before:
```
Total run time: 9.956319 secs
iteration process real start_mem_bytes end_mem_bytes
1 1 2.979338342 0.785497665 2493300736 3698921472
2 2 2.886093707 0.766516447 3699429376 3714109440
3 3 2.920944454 0.776514053 3714109440 3714109440
4 4 2.877046089 0.766922712 3714109440 3714109440
5 5 2.896827784 0.770490646 3714109440 3714109440
6 1 18.316171634 4.632888794 2493300736 3777204224
7 2 17.657202597 4.614435434 3731472384 3815968768
8 3 17.561911295 4.556154490 3744194560 3779891200
9 4 17.775040099 4.648452759 3744194560 3815968768
10 5 17.586604237 4.563611269 3744194560 3779891200
11 1 0.964031791 0.582021236 2493296640 3450097664
12 2 0.771914299 0.414866209 3342176256 3463372800
13 3 0.770634774 0.410928965 3347357696 3527225344
14 4 0.798272907 0.439264059 3352600576 3523477504
15 5 0.795549228 0.436603546 3357843456 3523641344
16 1 0.488568257 0.068626642 2493296640 3346874368
17 2 0.001445519 0.001448631 3346874368 3346874368
18 3 0.001467639 0.001470804 3346874368 3346874368
19 4 0.001425503 0.001428366 3346874368 3346874368
20 5 0.001460951 0.001464367 3346874368 3346874368
max_mem_bytes gc_level0 gc_level1 gc_level2 query cpu_count
1 3720196096 1 0 0 vignette 4
2 3720196096 0 0 0 vignette 4
3 3720196096 0 0 0 vignette 4
4 3720196096 0 0 0 vignette 4
5 3720196096 0 0 0 vignette 4
6 3796340736 2 0 1 payment_type_3 4
7 3815968768 1 0 1 payment_type_3 4
8 3815968768 1 0 0 payment_type_3 4
9 3815968768 0 0 1 payment_type_3 4
10 3815968768 1 0 0 payment_type_3 4
11 3481989120 2 1 3 small_no_files 4
12 3513794560 2 0 1 small_no_files 4
13 3527225344 1 0 1 small_no_files 4
14 3527225344 2 0 1 small_no_files 4
15 3527225344 0 1 1 small_no_files 4
16 3346874368 0 0 0 count_rows 4
17 3346874368 0 0 0 count_rows 4
18 3346874368 0 0 0 count_rows 4
19 3346874368 0 0 0 count_rows 4
20 3346874368 0 0 0 count_rows 4
```
After:
```
Total run time: 9.995613 secs
iteration process real start_mem_bytes end_mem_bytes
1 1 3.097148086 0.810223818 2493313024 3759751168
2 2 2.987976486 0.778924227 3760332800 3779207168
3 3 2.969162397 0.776125669 3779207168 3779207168
4 4 2.969802084 0.777535439 3779207168 3779207168
5 5 2.967313487 0.775563955 3779207168 3779207168
6 1 18.042212503 4.552006245 2493313024 3805003776
7 2 17.391559404 4.544502735 3759271936 3832758272
8 3 17.281636477 4.486674547 3760984064 3796680704
9 4 17.363466488 4.545516968 3760984064 3832758272
10 5 17.310964707 4.487994194 3760984064 3796680704
11 1 1.036147931 0.602855206 2493313024 3547631616
12 2 0.797990419 0.419062376 3439710208 3601801216
13 3 0.793205374 0.414812803 3485786112 3683479552
14 4 0.801588050 0.428945065 3508854784 3674488832
15 5 0.834174869 0.443217278 3508854784 3674656768
16 1 0.500276311 0.069972277 2493321216 3346898944
17 2 0.001478722 0.001481056 3346898944 3346898944
18 3 0.001478401 0.001481295 3346898944 3346898944
19 4 0.001472831 0.001475811 3346898944 3346898944
20 5 0.001461316 0.001464128 3346898944 3346898944
max_mem_bytes gc_level0 gc_level1 gc_level2 query cpu_count
1 3759751168 1 0 0 vignette 4
2 3779207168 0 0 0 vignette 4
3 3779207168 0 0 0 vignette 4
4 3779207168 0 0 0 vignette 4
5 3779207168 0 0 0 vignette 4
6 3824140288 2 0 1 payment_type_3 4
7 3832758272 1 0 1 payment_type_3 4
8 3832758272 1 0 0 payment_type_3 4
9 3832758272 0 0 1 payment_type_3 4
10 3832758272 1 0 0 payment_type_3 4
11 3579523072 2 1 3 small_no_files 4
12 3652222976 2 0 1 small_no_files 4
13 3683479552 1 0 1 small_no_files 4
14 3683479552 2 0 1 small_no_files 4
15 3683479552 0 1 1 small_no_files 4
16 3414007808 0 0 0 count_rows 4
17 3414007808 0 0 0 count_rows 4
18 3414007808 0 0 0 count_rows 4
19 3414007808 0 0 0 count_rows 4
20 3414007808 0 0 0 count_rows 4
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-933407083
@ursabot please benchmark name=file-read lang=Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-924887696
This helps quite a bit on all the [partitioned-dataset-filter](https://conbench.ursa.dev/compare/batches/3d25d19e1b2a46468bd5ddf5e82946e2...e0751834febe4a5c93fecd597faa1986/) benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-923135823
It appears this helps a bit in some benchmarks but hurts the taxi-parquet benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#discussion_r712604377
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -1116,6 +1116,151 @@ AsyncGenerator<T> MakeConcatenatedGenerator(AsyncGenerator<AsyncGenerator<T>> so
return MergedGenerator<T>(std::move(source), 1);
}
+/// \brief See MakeConcatenatedVectorGenerator
+template <typename T>
+class ConcatenatedVectorGenerator {
+ public:
+ explicit ConcatenatedVectorGenerator(
+ AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : state_(std::make_shared<State>(std::move(inner))) {}
+
+ Future<T> operator()() { return state_->Pull(); }
+
+ private:
+ struct State : public std::enable_shared_from_this<State> {
+ explicit State(AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : inner(std::move(inner)) {}
+
+ Future<T> Pull() {
+ auto guard = mutex.Lock();
+ if (finished) {
+ if (!error.ok()) {
+ return std::move(error);
+ }
+ return IterationTraits<T>::End();
+ }
+ if (available.empty()) {
+ auto waiting_future = Future<T>::Make();
+ waiting.push_back(waiting_future);
+
+ if (!waiting_for_results) {
+ waiting_for_results = true;
+ auto fut = inner();
+ // If fut is already complete, this may end up running the callback immediately,
+ // so unlock first
+ guard.Unlock();
+ auto self = this->shared_from_this();
+ fut.AddCallback(
+ [self, fut](const Result<util::optional<std::vector<T>>>&) mutable {
+ auto result = fut.MoveResult();
+ self->ProcessResult(std::move(result));
Review comment:
```suggestion
self->ProcessResult(fut.MoveResult());
```
Can you combine these two lines? Also, is the `fut` capture just to avoid the result copy?
##########
File path: cpp/src/arrow/dataset/file_parquet.cc
##########
@@ -489,9 +489,13 @@ Result<RecordBatchGenerator> ParquetFileFormat::ScanBatchesAsync(
auto parquet_scan_options,
GetFragmentScanOptions<ParquetFragmentScanOptions>(
kParquetTypeName, options.get(), default_fragment_scan_options));
- ARROW_ASSIGN_OR_RAISE(auto generator, reader->GetRecordBatchGenerator(
- reader, row_groups, column_projection,
- ::arrow::internal::GetCpuThreadPool()));
+ // Assume 1 row group corresponds to 1 batch (this factor could be
+ // improved by looking at metadata)
+ int row_group_readahead = options->batch_readahead;
+ ARROW_ASSIGN_OR_RAISE(
+ auto generator, reader->GetRecordBatchGenerator(
+ reader, row_groups, column_projection,
+ ::arrow::internal::GetCpuThreadPool(), row_group_readahead));
return MakeReadaheadGenerator(std::move(generator), options->batch_readahead);
Review comment:
Is this redundant at this point since you apply the row group readahead in the file?
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -1116,6 +1116,151 @@ AsyncGenerator<T> MakeConcatenatedGenerator(AsyncGenerator<AsyncGenerator<T>> so
return MergedGenerator<T>(std::move(source), 1);
}
+/// \brief See MakeConcatenatedVectorGenerator
+template <typename T>
+class ConcatenatedVectorGenerator {
+ public:
+ explicit ConcatenatedVectorGenerator(
+ AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : state_(std::make_shared<State>(std::move(inner))) {}
+
+ Future<T> operator()() { return state_->Pull(); }
+
+ private:
+ struct State : public std::enable_shared_from_this<State> {
+ explicit State(AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : inner(std::move(inner)) {}
+
+ Future<T> Pull() {
+ auto guard = mutex.Lock();
+ if (finished) {
+ if (!error.ok()) {
+ return std::move(error);
+ }
+ return IterationTraits<T>::End();
+ }
+ if (available.empty()) {
+ auto waiting_future = Future<T>::Make();
+ waiting.push_back(waiting_future);
+
+ if (!waiting_for_results) {
+ waiting_for_results = true;
+ auto fut = inner();
+ // If fut is already complete, this may end up running the callback immediately,
+ // so unlock first
+ guard.Unlock();
+ auto self = this->shared_from_this();
+ fut.AddCallback(
+ [self, fut](const Result<util::optional<std::vector<T>>>&) mutable {
+ auto result = fut.MoveResult();
+ self->ProcessResult(std::move(result));
+ });
+ }
+ return waiting_future;
+ }
+ assert(waiting.size() == 0);
+ auto result = Future<T>::MakeFinished(available.front());
+ available.pop_front();
+ return result;
+ }
+
+ void Purge() {
+ while (!waiting.empty()) {
+ waiting.front().MarkFinished(IterationTraits<T>::End());
+ waiting.pop_front();
+ }
+ }
+
+ void ProcessResult(Result<util::optional<std::vector<T>>> maybe_result) {
+ auto lock = mutex.Lock();
+ waiting_for_results = false;
+ // We should never call MarkFinished while holding the lock as a
+ // callback may call back into this generator
+ if (!maybe_result.ok()) {
+ finished = true;
+ if (waiting.empty()) {
+ error = maybe_result.status();
+ lock.Unlock();
+ } else {
+ lock.Unlock();
+ waiting.front().MarkFinished(maybe_result.status());
+ waiting.pop_front();
+ }
+ Purge();
+ return;
+ }
+
+ util::optional<std::vector<T>> result = maybe_result.MoveValueUnsafe();
+ if (!result.has_value()) {
+ finished = true;
+ lock.Unlock();
+ Purge();
+ return;
+ }
+
+ // MarkFinished might run a callback that tries to pull from us,
+ // so just record what needs finishing, and complete the futures
+ // later
+ std::vector<T> delivered = std::move(*result);
+ std::vector<Future<T>> deliver_to;
+ auto it = delivered.begin();
+ while (it != delivered.end() && !waiting.empty()) {
+ deliver_to.push_back(waiting.front());
+ waiting.pop_front();
+ it++;
+ }
+ available.insert(available.end(), std::make_move_iterator(it),
+ std::make_move_iterator(delivered.end()));
+
+ // If there's still more waiting futures, poll the source again
+ Future<util::optional<std::vector<T>>> fut;
+ waiting_for_results = !waiting.empty();
+ if (waiting_for_results) {
+ fut = inner();
+ }
+
+ lock.Unlock();
+ // Now actually MarkFinished
+ auto next = delivered.begin();
+ for (auto& fut : deliver_to) {
+ fut.MarkFinished(std::move(*next));
+ next++;
+ }
+
+ if (!fut.is_valid()) return;
+ // Must do this outside the lock since if fut is already complete, this callback
+ // will run right away
+ auto self = this->shared_from_this();
+ fut.AddCallback([self, fut](const Result<util::optional<std::vector<T>>>&) mutable {
+ auto result = fut.MoveResult();
+ self->ProcessResult(std::move(result));
+ });
+ }
+
+ AsyncGenerator<util::optional<std::vector<T>>> inner;
+ util::Mutex mutex;
+ std::deque<Future<T>> waiting;
+ std::deque<T> available;
+ Status error;
+ // Are we finished polling the source?
+ bool finished = false;
+ // Do we have a request in flight for the source?
+ bool waiting_for_results = false;
+ };
+
+ std::shared_ptr<State> state_;
+};
+
+/// \brief Creates a generator that converts a stream of vector<T> to a stream of T.
+///
+/// This generator is async-reentrant but will never pull from source reentrantly.
+/// Instead, combine with MakeReadaheadGenerator.
Review comment:
Is this much different than `MakeConcatenatedGenerator(MakeMappedGenerator(generator_of_vectors, MakeVectorGenerator))?`
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -1116,6 +1116,151 @@ AsyncGenerator<T> MakeConcatenatedGenerator(AsyncGenerator<AsyncGenerator<T>> so
return MergedGenerator<T>(std::move(source), 1);
}
+/// \brief See MakeConcatenatedVectorGenerator
+template <typename T>
+class ConcatenatedVectorGenerator {
+ public:
+ explicit ConcatenatedVectorGenerator(
+ AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : state_(std::make_shared<State>(std::move(inner))) {}
+
+ Future<T> operator()() { return state_->Pull(); }
+
+ private:
+ struct State : public std::enable_shared_from_this<State> {
+ explicit State(AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : inner(std::move(inner)) {}
+
+ Future<T> Pull() {
+ auto guard = mutex.Lock();
+ if (finished) {
+ if (!error.ok()) {
+ return std::move(error);
+ }
+ return IterationTraits<T>::End();
+ }
+ if (available.empty()) {
+ auto waiting_future = Future<T>::Make();
+ waiting.push_back(waiting_future);
+
+ if (!waiting_for_results) {
+ waiting_for_results = true;
+ auto fut = inner();
+ // If fut is already complete, this may end up running the callback immediately,
+ // so unlock first
+ guard.Unlock();
+ auto self = this->shared_from_this();
+ fut.AddCallback(
+ [self, fut](const Result<util::optional<std::vector<T>>>&) mutable {
+ auto result = fut.MoveResult();
+ self->ProcessResult(std::move(result));
+ });
+ }
+ return waiting_future;
+ }
+ assert(waiting.size() == 0);
+ auto result = Future<T>::MakeFinished(available.front());
+ available.pop_front();
+ return result;
+ }
+
+ void Purge() {
+ while (!waiting.empty()) {
+ waiting.front().MarkFinished(IterationTraits<T>::End());
+ waiting.pop_front();
+ }
+ }
+
+ void ProcessResult(Result<util::optional<std::vector<T>>> maybe_result) {
+ auto lock = mutex.Lock();
+ waiting_for_results = false;
+ // We should never call MarkFinished while holding the lock as a
+ // callback may call back into this generator
+ if (!maybe_result.ok()) {
+ finished = true;
+ if (waiting.empty()) {
Review comment:
No harm in having this case but I'm not sure if `waiting` can be empty here.
##########
File path: cpp/src/arrow/util/async_generator_test.cc
##########
@@ -461,6 +461,68 @@ TEST(TestAsyncUtil, Concatenated) {
AssertAsyncGeneratorMatch(expected, concat);
}
+TEST(TestAsyncUtil, ConcatenatedVector) {
Review comment:
Given the locks you have it would be good to have at least one parallel test using a readahead generator.
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -1116,6 +1116,151 @@ AsyncGenerator<T> MakeConcatenatedGenerator(AsyncGenerator<AsyncGenerator<T>> so
return MergedGenerator<T>(std::move(source), 1);
}
+/// \brief See MakeConcatenatedVectorGenerator
+template <typename T>
+class ConcatenatedVectorGenerator {
+ public:
+ explicit ConcatenatedVectorGenerator(
+ AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : state_(std::make_shared<State>(std::move(inner))) {}
+
+ Future<T> operator()() { return state_->Pull(); }
+
+ private:
+ struct State : public std::enable_shared_from_this<State> {
+ explicit State(AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : inner(std::move(inner)) {}
+
+ Future<T> Pull() {
+ auto guard = mutex.Lock();
+ if (finished) {
+ if (!error.ok()) {
+ return std::move(error);
+ }
+ return IterationTraits<T>::End();
+ }
+ if (available.empty()) {
+ auto waiting_future = Future<T>::Make();
+ waiting.push_back(waiting_future);
+
+ if (!waiting_for_results) {
+ waiting_for_results = true;
+ auto fut = inner();
+ // If fut is already complete, this may end up running the callback immediately,
+ // so unlock first
+ guard.Unlock();
+ auto self = this->shared_from_this();
+ fut.AddCallback(
+ [self, fut](const Result<util::optional<std::vector<T>>>&) mutable {
+ auto result = fut.MoveResult();
+ self->ProcessResult(std::move(result));
+ });
+ }
+ return waiting_future;
+ }
+ assert(waiting.size() == 0);
+ auto result = Future<T>::MakeFinished(available.front());
+ available.pop_front();
+ return result;
+ }
+
+ void Purge() {
+ while (!waiting.empty()) {
+ waiting.front().MarkFinished(IterationTraits<T>::End());
+ waiting.pop_front();
+ }
+ }
+
+ void ProcessResult(Result<util::optional<std::vector<T>>> maybe_result) {
+ auto lock = mutex.Lock();
+ waiting_for_results = false;
+ // We should never call MarkFinished while holding the lock as a
+ // callback may call back into this generator
+ if (!maybe_result.ok()) {
+ finished = true;
+ if (waiting.empty()) {
+ error = maybe_result.status();
+ lock.Unlock();
+ } else {
+ lock.Unlock();
+ waiting.front().MarkFinished(maybe_result.status());
+ waiting.pop_front();
+ }
+ Purge();
+ return;
+ }
+
+ util::optional<std::vector<T>> result = maybe_result.MoveValueUnsafe();
+ if (!result.has_value()) {
+ finished = true;
+ lock.Unlock();
+ Purge();
+ return;
+ }
+
+ // MarkFinished might run a callback that tries to pull from us,
+ // so just record what needs finishing, and complete the futures
+ // later
+ std::vector<T> delivered = std::move(*result);
+ std::vector<Future<T>> deliver_to;
+ auto it = delivered.begin();
+ while (it != delivered.end() && !waiting.empty()) {
+ deliver_to.push_back(waiting.front());
+ waiting.pop_front();
+ it++;
+ }
+ available.insert(available.end(), std::make_move_iterator(it),
+ std::make_move_iterator(delivered.end()));
+
+ // If there's still more waiting futures, poll the source again
+ Future<util::optional<std::vector<T>>> fut;
+ waiting_for_results = !waiting.empty();
+ if (waiting_for_results) {
+ fut = inner();
+ }
+
+ lock.Unlock();
+ // Now actually MarkFinished
+ auto next = delivered.begin();
+ for (auto& fut : deliver_to) {
+ fut.MarkFinished(std::move(*next));
+ next++;
+ }
+
+ if (!fut.is_valid()) return;
+ // Must do this outside the lock since if fut is already complete, this callback
+ // will run right away
+ auto self = this->shared_from_this();
+ fut.AddCallback([self, fut](const Result<util::optional<std::vector<T>>>&) mutable {
+ auto result = fut.MoveResult();
+ self->ProcessResult(std::move(result));
+ });
+ }
+
+ AsyncGenerator<util::optional<std::vector<T>>> inner;
+ util::Mutex mutex;
+ std::deque<Future<T>> waiting;
+ std::deque<T> available;
+ Status error;
+ // Are we finished polling the source?
+ bool finished = false;
+ // Do we have a request in flight for the source?
+ bool waiting_for_results = false;
+ };
+
+ std::shared_ptr<State> state_;
+};
+
+/// \brief Creates a generator that converts a stream of vector<T> to a stream of T.
+///
+/// This generator is async-reentrant but will never pull from source reentrantly.
+/// Instead, combine with MakeReadaheadGenerator.
+template <typename T>
+AsyncGenerator<T> MakeConcatenatedVectorGenerator(
Review comment:
Maybe `MakeFlattenGenerator` to match `MakeFlattenIterator`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #11189:
URL: https://github.com/apache/arrow/pull/11189
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-926041334
Moving to draft for now since this needs the same fixes as #11207.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-922990928
https://issues.apache.org/jira/browse/ARROW-14026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-924300963
Benchmark runs are scheduled for baseline = 18756c2aa60344f9eb5b896ff2cee4e734b92516 and contender = 301e86505a0b2b27a2d6fa7d446e2aa1a136c9ce. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Provided benchmark filters do not have any benchmark groups to be executed on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/0ec8ed37a55c4440bc0f7d3ffdb491f8...4610757ce5e64be5a0237b0f2a037b9b/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/f9a453ffd2e14e10a74f3e7c71ddfb77...61f200d18df143259815b85175a790ab/)
[Skipped :warning: Only ['C++', 'Java'] langs are supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/a9ad001bb40e41808430a974c869499e...7a618af070b149e1bfaec737bf21eb69/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-922991831
@ursabot please benchmark lang=R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-924300191
@ursabot please benchmark lang=R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-933408349
Supported benchmark command examples:
`@ursabot benchmark help`
To run all benchmarks:
`@ursabot please benchmark`
To filter benchmarks by language:
`@ursabot please benchmark lang=Python`
`@ursabot please benchmark lang=C++`
`@ursabot please benchmark lang=R`
`@ursabot please benchmark lang=Java`
`@ursabot please benchmark lang=JavaScript`
To filter Python and R benchmarks by name:
`@ursabot please benchmark name=file-write`
`@ursabot please benchmark name=file-write lang=Python`
`@ursabot please benchmark name=file-.*`
To filter C++ benchmarks by archery --suite-filter and --benchmark-filter:
`@ursabot please benchmark command=cpp-micro --suite-filter=arrow-compute-vector-selection-benchmark --benchmark-filter=TakeStringRandomIndicesWithNulls/262144/2 --iterations=3`
For other `command=cpp-micro` options, please see https://github.com/ursacomputing/benchmarks/blob/main/benchmarks/cpp_micro_benchmarks.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-927985770
@westonpace any other comments here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-924300963
Benchmark runs are scheduled for baseline = 18756c2aa60344f9eb5b896ff2cee4e734b92516 and contender = 301e86505a0b2b27a2d6fa7d446e2aa1a136c9ce. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Provided benchmark filters do not have any benchmark groups to be executed on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/0ec8ed37a55c4440bc0f7d3ffdb491f8...4610757ce5e64be5a0237b0f2a037b9b/)
[Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/f9a453ffd2e14e10a74f3e7c71ddfb77...61f200d18df143259815b85175a790ab/)
[Skipped :warning: Only ['C++', 'Java'] langs are supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/a9ad001bb40e41808430a974c869499e...7a618af070b149e1bfaec737bf21eb69/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-933408336
@ursabot help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot edited a comment on pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#issuecomment-922992268
Benchmark runs are scheduled for baseline = 7128b5eb4302a768e48e0a4718b445495fc3e107 and contender = dc36f09c16a2933614b41f8912968cf1cfb364d8. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Provided benchmark filters do not have any benchmark groups to be executed on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/14909b12ff3b44afb6c2be83853db45b...deb4ec4f85ba4cab97cdc36a4c2d5d9b/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/f7dbbd8836ac43f69a52b17caa64dd26...f0e710cddcef40e89344f39f2dd0e6ba/)
[Skipped :warning: Only ['C++', 'Java'] langs are supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/2cb2668a5e37451f9d68770ce2a12a09...16fc2e6d8ec346679826ead5c66f42b7/)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#discussion_r712967234
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -1116,6 +1116,151 @@ AsyncGenerator<T> MakeConcatenatedGenerator(AsyncGenerator<AsyncGenerator<T>> so
return MergedGenerator<T>(std::move(source), 1);
}
+/// \brief See MakeConcatenatedVectorGenerator
+template <typename T>
+class ConcatenatedVectorGenerator {
+ public:
+ explicit ConcatenatedVectorGenerator(
+ AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : state_(std::make_shared<State>(std::move(inner))) {}
+
+ Future<T> operator()() { return state_->Pull(); }
+
+ private:
+ struct State : public std::enable_shared_from_this<State> {
+ explicit State(AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : inner(std::move(inner)) {}
+
+ Future<T> Pull() {
+ auto guard = mutex.Lock();
+ if (finished) {
+ if (!error.ok()) {
+ return std::move(error);
+ }
+ return IterationTraits<T>::End();
+ }
+ if (available.empty()) {
+ auto waiting_future = Future<T>::Make();
+ waiting.push_back(waiting_future);
+
+ if (!waiting_for_results) {
+ waiting_for_results = true;
+ auto fut = inner();
+ // If fut is already complete, this may end up running the callback immediately,
+ // so unlock first
+ guard.Unlock();
+ auto self = this->shared_from_this();
+ fut.AddCallback(
+ [self, fut](const Result<util::optional<std::vector<T>>>&) mutable {
+ auto result = fut.MoveResult();
+ self->ProcessResult(std::move(result));
+ });
+ }
+ return waiting_future;
+ }
+ assert(waiting.size() == 0);
+ auto result = Future<T>::MakeFinished(available.front());
+ available.pop_front();
+ return result;
+ }
+
+ void Purge() {
+ while (!waiting.empty()) {
+ waiting.front().MarkFinished(IterationTraits<T>::End());
+ waiting.pop_front();
+ }
+ }
+
+ void ProcessResult(Result<util::optional<std::vector<T>>> maybe_result) {
+ auto lock = mutex.Lock();
+ waiting_for_results = false;
+ // We should never call MarkFinished while holding the lock as a
+ // callback may call back into this generator
+ if (!maybe_result.ok()) {
+ finished = true;
+ if (waiting.empty()) {
+ error = maybe_result.status();
+ lock.Unlock();
+ } else {
+ lock.Unlock();
+ waiting.front().MarkFinished(maybe_result.status());
+ waiting.pop_front();
+ }
+ Purge();
+ return;
+ }
+
+ util::optional<std::vector<T>> result = maybe_result.MoveValueUnsafe();
+ if (!result.has_value()) {
+ finished = true;
+ lock.Unlock();
+ Purge();
+ return;
+ }
+
+ // MarkFinished might run a callback that tries to pull from us,
+ // so just record what needs finishing, and complete the futures
+ // later
+ std::vector<T> delivered = std::move(*result);
+ std::vector<Future<T>> deliver_to;
+ auto it = delivered.begin();
+ while (it != delivered.end() && !waiting.empty()) {
+ deliver_to.push_back(waiting.front());
+ waiting.pop_front();
+ it++;
+ }
+ available.insert(available.end(), std::make_move_iterator(it),
+ std::make_move_iterator(delivered.end()));
+
+ // If there's still more waiting futures, poll the source again
+ Future<util::optional<std::vector<T>>> fut;
+ waiting_for_results = !waiting.empty();
+ if (waiting_for_results) {
+ fut = inner();
+ }
+
+ lock.Unlock();
+ // Now actually MarkFinished
+ auto next = delivered.begin();
+ for (auto& fut : deliver_to) {
+ fut.MarkFinished(std::move(*next));
+ next++;
+ }
+
+ if (!fut.is_valid()) return;
+ // Must do this outside the lock since if fut is already complete, this callback
+ // will run right away
+ auto self = this->shared_from_this();
+ fut.AddCallback([self, fut](const Result<util::optional<std::vector<T>>>&) mutable {
+ auto result = fut.MoveResult();
+ self->ProcessResult(std::move(result));
+ });
+ }
+
+ AsyncGenerator<util::optional<std::vector<T>>> inner;
+ util::Mutex mutex;
+ std::deque<Future<T>> waiting;
+ std::deque<T> available;
+ Status error;
+ // Are we finished polling the source?
+ bool finished = false;
+ // Do we have a request in flight for the source?
+ bool waiting_for_results = false;
+ };
+
+ std::shared_ptr<State> state_;
+};
+
+/// \brief Creates a generator that converts a stream of vector<T> to a stream of T.
+///
+/// This generator is async-reentrant but will never pull from source reentrantly.
+/// Instead, combine with MakeReadaheadGenerator.
Review comment:
Actually, now that you point that out…I suppose there's one layer of indirection that this gets rid of (once the next vector arrives, any pending futures will get fulfilled right away, instead of going through the callbacks. But I'll check if this actually affects performance/make sure we get parallelism, if not, we should just use the existing ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #11189: ARROW-14026: [C++] Enable batch parallelism in Parquet scanner
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11189:
URL: https://github.com/apache/arrow/pull/11189#discussion_r713334535
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -1116,6 +1116,151 @@ AsyncGenerator<T> MakeConcatenatedGenerator(AsyncGenerator<AsyncGenerator<T>> so
return MergedGenerator<T>(std::move(source), 1);
}
+/// \brief See MakeConcatenatedVectorGenerator
+template <typename T>
+class ConcatenatedVectorGenerator {
+ public:
+ explicit ConcatenatedVectorGenerator(
+ AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : state_(std::make_shared<State>(std::move(inner))) {}
+
+ Future<T> operator()() { return state_->Pull(); }
+
+ private:
+ struct State : public std::enable_shared_from_this<State> {
+ explicit State(AsyncGenerator<util::optional<std::vector<T>>> inner)
+ : inner(std::move(inner)) {}
+
+ Future<T> Pull() {
+ auto guard = mutex.Lock();
+ if (finished) {
+ if (!error.ok()) {
+ return std::move(error);
+ }
+ return IterationTraits<T>::End();
+ }
+ if (available.empty()) {
+ auto waiting_future = Future<T>::Make();
+ waiting.push_back(waiting_future);
+
+ if (!waiting_for_results) {
+ waiting_for_results = true;
+ auto fut = inner();
+ // If fut is already complete, this may end up running the callback immediately,
+ // so unlock first
+ guard.Unlock();
+ auto self = this->shared_from_this();
+ fut.AddCallback(
+ [self, fut](const Result<util::optional<std::vector<T>>>&) mutable {
+ auto result = fut.MoveResult();
+ self->ProcessResult(std::move(result));
+ });
+ }
+ return waiting_future;
+ }
+ assert(waiting.size() == 0);
+ auto result = Future<T>::MakeFinished(available.front());
+ available.pop_front();
+ return result;
+ }
+
+ void Purge() {
+ while (!waiting.empty()) {
+ waiting.front().MarkFinished(IterationTraits<T>::End());
+ waiting.pop_front();
+ }
+ }
+
+ void ProcessResult(Result<util::optional<std::vector<T>>> maybe_result) {
+ auto lock = mutex.Lock();
+ waiting_for_results = false;
+ // We should never call MarkFinished while holding the lock as a
+ // callback may call back into this generator
+ if (!maybe_result.ok()) {
+ finished = true;
+ if (waiting.empty()) {
+ error = maybe_result.status();
+ lock.Unlock();
+ } else {
+ lock.Unlock();
+ waiting.front().MarkFinished(maybe_result.status());
+ waiting.pop_front();
+ }
+ Purge();
+ return;
+ }
+
+ util::optional<std::vector<T>> result = maybe_result.MoveValueUnsafe();
+ if (!result.has_value()) {
+ finished = true;
+ lock.Unlock();
+ Purge();
+ return;
+ }
+
+ // MarkFinished might run a callback that tries to pull from us,
+ // so just record what needs finishing, and complete the futures
+ // later
+ std::vector<T> delivered = std::move(*result);
+ std::vector<Future<T>> deliver_to;
+ auto it = delivered.begin();
+ while (it != delivered.end() && !waiting.empty()) {
+ deliver_to.push_back(waiting.front());
+ waiting.pop_front();
+ it++;
+ }
+ available.insert(available.end(), std::make_move_iterator(it),
+ std::make_move_iterator(delivered.end()));
+
+ // If there's still more waiting futures, poll the source again
+ Future<util::optional<std::vector<T>>> fut;
+ waiting_for_results = !waiting.empty();
+ if (waiting_for_results) {
+ fut = inner();
+ }
+
+ lock.Unlock();
+ // Now actually MarkFinished
+ auto next = delivered.begin();
+ for (auto& fut : deliver_to) {
+ fut.MarkFinished(std::move(*next));
+ next++;
+ }
+
+ if (!fut.is_valid()) return;
+ // Must do this outside the lock since if fut is already complete, this callback
+ // will run right away
+ auto self = this->shared_from_this();
+ fut.AddCallback([self, fut](const Result<util::optional<std::vector<T>>>&) mutable {
+ auto result = fut.MoveResult();
+ self->ProcessResult(std::move(result));
+ });
+ }
+
+ AsyncGenerator<util::optional<std::vector<T>>> inner;
+ util::Mutex mutex;
+ std::deque<Future<T>> waiting;
+ std::deque<T> available;
+ Status error;
+ // Are we finished polling the source?
+ bool finished = false;
+ // Do we have a request in flight for the source?
+ bool waiting_for_results = false;
+ };
+
+ std::shared_ptr<State> state_;
+};
+
+/// \brief Creates a generator that converts a stream of vector<T> to a stream of T.
+///
+/// This generator is async-reentrant but will never pull from source reentrantly.
+/// Instead, combine with MakeReadaheadGenerator.
Review comment:
Alright, it seems to be about the same, so I went with that solution - saves us both work! (This is effectively what you commented in JIRA: we just implement batch readahead as row group readahead.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org