You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/21 20:44:34 UTC
[GitHub] [arrow] lidavidm opened a new pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
lidavidm opened a new pull request #11207:
URL: https://github.com/apache/arrow/pull/11207
This wraps the IPC/CSV generators in a sub-generator that will re-slice the batches to respect the batch_size parameter. For Parquet, see #11189.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #11207:
URL: https://github.com/apache/arrow/pull/11207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#issuecomment-925251704
@westonpace there are some very fun edge cases I ran into with the generators here, let me know what you think of the workarounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#discussion_r714286753
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -221,6 +221,9 @@ class MappingGenerator {
bool should_trigger;
{
auto guard = state->mutex.Lock();
+ // A MappedCallback may have purged or be purging the queue;
+ // we shouldn't do anything here.
+ if (state->finished) return;
Review comment:
Shoot, I kind of forgot about this but I have this fix in my CSV streaming PR I kind of half-abandoned because it was too complex and I needed to work on other things. Can you grab the QueuingMapFailStress test case from https://github.com/apache/arrow/pull/10795/files#diff-aa924daf1e09b13ad031b70e40eeff3b4d4b817b2a32981bb0c0fd2e5313326cR500
I like your fix better anyways. Instead of returning early I just added more flags and ifs which was a more complex version of the same thing.
##########
File path: cpp/src/arrow/dataset/dataset_internal.h
##########
@@ -132,5 +133,28 @@ class FragmentDataset : public Dataset {
AsyncGenerator<std::shared_ptr<Fragment>> fragment_gen_;
};
+// Given a record batch generator, creates a new generator that slices
+// batches so individual batches have at most batch_size rows. The
+// resulting generator is async-reentrant, but does not forward
+// reentrant pulls, so apply readahead before using this helper.
+inline RecordBatchGenerator MakeChunkedBatchGenerator(RecordBatchGenerator gen,
+ int64_t batch_size) {
+ return MakeConcatenatedGenerator(MakeMappedGenerator(
+ std::move(gen),
+ [batch_size](const std::shared_ptr<RecordBatch>& batch)
+ -> ::arrow::AsyncGenerator<std::shared_ptr<::arrow::RecordBatch>> {
+ const int64_t rows = batch->num_rows();
+ if (rows <= batch_size) {
+ return ::arrow::MakeVectorGenerator<std::shared_ptr<RecordBatch>>({batch});
+ }
+ std::vector<std::shared_ptr<RecordBatch>> slices;
+ slices.reserve(rows / batch_size + (rows % batch_size != 0));
+ for (int64_t i = 0; i < rows; i += batch_size) {
+ slices.push_back(batch->Slice(i, batch_size));
+ }
+ return ::arrow::MakeVectorGenerator(std::move(slices));
Review comment:
I wonder if we want to create a simple helper function MakeFlattenGenerator which just calls MakeConcatenated(MakeMapped) under the hood. That way if we ever do decide the overhead is too much we can change the implementation.
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -1001,32 +1004,45 @@ class MergedGenerator {
};
struct InnerCallback {
Review comment:
Ok, so let me make sure I understand the change. If a merged generator pulled off a generator and that generator had a whole bunch of already completed futures (e.g. vector generator) then we would end up recursing for each call into the child.
Instead, you setup the TryAddCallback loop to only recurse when the future isn't finished. I think this is a great addition. Even if we weren't overflowing the stack in most cases we'd still be creating unnecessarily long stack traces. Thanks for working through this.
##########
File path: cpp/src/arrow/dataset/test_util.h
##########
@@ -589,6 +589,23 @@ class FileFormatScanMixin : public FileFormatFixtureMixin<FormatHelper>,
}
ASSERT_EQ(row_count, GetParam().expected_rows());
}
+ // Ensure batch_size is respected
+ void TestScanBatchSize() {
+ auto reader = GetRecordBatchReader(schema({field("f64", float64())}));
+ auto source = this->GetFileSource(reader.get());
+
+ this->SetSchema(reader->schema()->fields());
+ auto fragment = this->MakeFragment(*source);
+
+ int64_t row_count = 0;
+ opts_->batch_size = 16;
Review comment:
I think all of our test cases have # rows evenly divisible by 16. Maybe make this 15 or something odd so that we ensure we test cases where the read doesn't split into even batches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#discussion_r714333823
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -221,6 +221,9 @@ class MappingGenerator {
bool should_trigger;
{
auto guard = state->mutex.Lock();
+ // A MappedCallback may have purged or be purging the queue;
+ // we shouldn't do anything here.
+ if (state->finished) return;
Review comment:
You can ignore the third parameter. Odd that the overload doesn't exist :confused:. It was basically to toggle between a newly added BasicMappingGenerator (which pulls from source reentrantly) and the regular mapping generator (which does not pull from the source reentrantly but does allow the map function to run in parallel). I think the test is always using `true` which maps to what we currently have. So just drop the `true` and you should be ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#discussion_r714326412
##########
File path: cpp/src/arrow/dataset/dataset_internal.h
##########
@@ -132,5 +133,28 @@ class FragmentDataset : public Dataset {
AsyncGenerator<std::shared_ptr<Fragment>> fragment_gen_;
};
+// Given a record batch generator, creates a new generator that slices
+// batches so individual batches have at most batch_size rows. The
+// resulting generator is async-reentrant, but does not forward
+// reentrant pulls, so apply readahead before using this helper.
+inline RecordBatchGenerator MakeChunkedBatchGenerator(RecordBatchGenerator gen,
+ int64_t batch_size) {
+ return MakeConcatenatedGenerator(MakeMappedGenerator(
+ std::move(gen),
+ [batch_size](const std::shared_ptr<RecordBatch>& batch)
+ -> ::arrow::AsyncGenerator<std::shared_ptr<::arrow::RecordBatch>> {
+ const int64_t rows = batch->num_rows();
+ if (rows <= batch_size) {
+ return ::arrow::MakeVectorGenerator<std::shared_ptr<RecordBatch>>({batch});
+ }
+ std::vector<std::shared_ptr<RecordBatch>> slices;
+ slices.reserve(rows / batch_size + (rows % batch_size != 0));
+ for (int64_t i = 0; i < rows; i += batch_size) {
+ slices.push_back(batch->Slice(i, batch_size));
+ }
+ return ::arrow::MakeVectorGenerator(std::move(slices));
Review comment:
Makes sense. This is basically flatmap so I've named it accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#issuecomment-924369751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#discussion_r714339906
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -221,6 +221,9 @@ class MappingGenerator {
bool should_trigger;
{
auto guard = state->mutex.Lock();
+ // A MappedCallback may have purged or be purging the queue;
+ // we shouldn't do anything here.
+ if (state->finished) return;
Review comment:
Ah I figured it would be something like that, I added the test here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#issuecomment-926040989
Once this lands I'll fix up #11189 since it uses the same fix underneath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#discussion_r714326580
##########
File path: cpp/src/arrow/dataset/test_util.h
##########
@@ -589,6 +589,23 @@ class FileFormatScanMixin : public FileFormatFixtureMixin<FormatHelper>,
}
ASSERT_EQ(row_count, GetParam().expected_rows());
}
+ // Ensure batch_size is respected
+ void TestScanBatchSize() {
+ auto reader = GetRecordBatchReader(schema({field("f64", float64())}));
+ auto source = this->GetFileSource(reader.get());
+
+ this->SetSchema(reader->schema()->fields());
+ auto fragment = this->MakeFragment(*source);
+
+ int64_t row_count = 0;
+ opts_->batch_size = 16;
Review comment:
Good point, I bumped it to 17 which is also prime!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#issuecomment-925044012
Hmm, I think the remaining failures are possibly because we're overflowing the stack - having SSH'd to a MacOS machine to look at things, we have > 2000 stack frames and lldb can't even print a backtrace without segfaulting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a change in pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#discussion_r714322177
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -221,6 +221,9 @@ class MappingGenerator {
bool should_trigger;
{
auto guard = state->mutex.Lock();
+ // A MappedCallback may have purged or be purging the queue;
+ // we shouldn't do anything here.
+ if (state->finished) return;
Review comment:
That test case uses an overload of MakeMappedGenerator which doesn't exist in the fork (it takes a bool as the third argument) - what was the purpose there?
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -1001,32 +1004,45 @@ class MergedGenerator {
};
struct InnerCallback {
Review comment:
Yup, that's it. And in particular the degenerate case here is that the synchronously-run callback from the mapped generator's futures would pull from the mapped generator, creating a loop implemented as (non-tail) recursion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #11207: ARROW-14024: [C++] Test that batch size is respected for IPC/CSV
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #11207:
URL: https://github.com/apache/arrow/pull/11207#issuecomment-925065761
Ah, most of the stack frames look like this, once you remove irrelevant ones:
```
2021-09-22T14:19:28.6454400Z frame #83: 0x0000000105789ea0 libarrow_dataset.600.0.0.dylib`arrow::MergedGenerator<arrow::dataset::EnumeratedRecordBatch>::InnerCallback::operator(this=0x00007fc747637bc8, maybe_next=0x00007fc74763cd20)(arrow::Result<arrow::dataset::EnumeratedRecordBatch> const&) at async_generator.h:1029:14
2021-09-22T14:19:28.6479720Z frame #91: 0x0000000105789f70 libarrow_dataset.600.0.0.dylib`arrow::MergedGenerator<arrow::dataset::EnumeratedRecordBatch>::InnerCallback::operator(this=0x00007fc74763d1a8, maybe_next=0x00007fc74763b8f0)(arrow::Result<arrow::dataset::EnumeratedRecordBatch> const&) at async_generator.h:1031:48
2021-09-22T14:19:28.6503520Z frame #99: 0x0000000105789f70 libarrow_dataset.600.0.0.dylib`arrow::MergedGenerator<arrow::dataset::EnumeratedRecordBatch>::InnerCallback::operator(this=0x00007fc7476357f8, maybe_next=0x00007fc74763cde0)(arrow::Result<arrow::dataset::EnumeratedRecordBatch> const&) at async_generator.h:1031:48
2021-09-22T14:19:28.6524850Z frame #107: 0x0000000105789f70 libarrow_dataset.600.0.0.dylib`arrow::MergedGenerator<arrow::dataset::EnumeratedRecordBatch>::InnerCallback::operator(this=0x00007fc74763c008, maybe_next=0x00007fc74763cc60)(arrow::Result<arrow::dataset::EnumeratedRecordBatch> const&) at async_generator.h:1031:48
2021-09-22T14:19:28.6552850Z frame #115: 0x0000000105789f70 libarrow_dataset.600.0.0.dylib`arrow::MergedGenerator<arrow::dataset::EnumeratedRecordBatch>::InnerCallback::operator(this=0x00007fc74747fec8, maybe_next=0x00007fc74747a480)(arrow::Result<arrow::dataset::EnumeratedRecordBatch> const&) at async_generator.h:1031:48
2021-09-22T14:19:28.6580800Z frame #123: 0x0000000105789f70 libarrow_dataset.600.0.0.dylib`arrow::MergedGenerator<arrow::dataset::EnumeratedRecordBatch>::InnerCallback::operator(this=0x00007fc7475919c8, maybe_next=0x00007fc74759b2a0)(arrow::Result<arrow::dataset::EnumeratedRecordBatch> const&) at async_generator.h:1031:48
```
I suppose what's going on is that the merged generator's inner generator is returning a completed future, so callbacks run immediately, and in the process another pull is made to the merged generator; then the merged generator's callback is run, and hence we get stuck in recursion (limited only by the size of the underlying generator). And this only affects the new test, which slices batches up, leading to there being 1024 batches, multiplied by ~10 or so frames per batch - causing a stack overflow. A similar issue happens if we drop the stack size on Linux.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org