[GitHub] [arrow] westonpace commented on a change in pull request #11285: ARROW-13611: [C++] Scanning datasets does not enforce back pressure

[GitBox <gi...@apache.org>]

westonpace commented on a change in pull request #11285:
URL: https://github.com/apache/arrow/pull/11285#discussion_r720449458



##########
File path: cpp/src/arrow/dataset/scanner.cc
##########
@@ -593,20 +593,23 @@ Result<EnumeratedRecordBatchGenerator> AsyncScanner::ScanBatchesUnorderedAsync(
   ARROW_ASSIGN_OR_RAISE(auto plan, compute::ExecPlan::Make(exec_context.get()));
   AsyncGenerator<util::optional<compute::ExecBatch>> sink_gen;
 
+  util::BackpressureOptions backpressure =
+      util::MakeBackpressureOptions(kDefaultBackpressureLow, kDefaultBackpressureHigh);

Review comment:
       I don't want to add too many tuning parameters to scan options.  I think I'd wait until these defaults don't work for someone before exposing them.  At the moment I couldn't give any reasonable advice on how to tune these.  Increasing the values would increase the amount of memory used but I don't believe it would have any significant impact on performance in most cases.  The backpressure limit in the dataset writer is also hidden from the user.  It might be that we want a single parameter to influence backpressure tuning settings across the board.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org