You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/02 14:28:08 UTC

[GitHub] [arrow] paleolimbot opened a new issue, #15151: Should `GetIOThreadPool()` be accessible from installed headers?

paleolimbot opened a new issue, #15151:
URL: https://github.com/apache/arrow/issues/15151

   ### Describe the enhancement requested
   
   In #14582 it was found that using the CPU thread pool in `arrow::compute::MakeReaderGenerator()` caused problems when the number of CPU threads was limited (as it often is on CI machines with few available cores). The solution was to use the IO thread pool for this; however, `arrow::io::internal::GetIOThreadPool()` is not available in any installed headers. I don't know what the best way to make this available would be (or whether creating a source node from a record batch reader should be baked into the internals somewhere); however, my hack of:
   
   ```r
   namespace arrow {
   namespace io {
   namespace internal {
   arrow::internal::ThreadPool* GetIOThreadPool();
   }
   }  // namespace io
   }  // namespace arrow
   ```
   
   ...in the R package should almost certainly not exist.
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] vibhatha commented on issue #15151: Should `GetIOThreadPool()` be accessible from installed headers?

Posted by GitBox <gi...@apache.org>.
vibhatha commented on issue #15151:
URL: https://github.com/apache/arrow/issues/15151#issuecomment-1369870067

   Thanks @paleolimbot, I will work on this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] vibhatha commented on issue #15151: Should `GetIOThreadPool()` be accessible from installed headers?

Posted by GitBox <gi...@apache.org>.
vibhatha commented on issue #15151:
URL: https://github.com/apache/arrow/issues/15151#issuecomment-1369374228

   > > As a short term hack you can do:
   > 
   > We do this already in the R package for the place where we need to use the IO thread pool to submit jobs...the problem here is that we need a `ThreadPool*` to pass to `MakeReaderGenerator()`.
   > 
   > > I think it would be a good addition to add `record_batch_reader_source`.
   > 
   > That would be my preferred solution...I'd rather not maintain the best way to do that in the R package and it's come up on the mailing list in a context unrelated to the R package as well ( https://lists.apache.org/thread/zo9qq0pntkrt2vnczoxx7hfsl6k233zy ).
   
   @paleolimbot The reference link [here](https://lists.apache.org/thread/zo9qq0pntkrt2vnczoxx7hfsl6k233zy) which refers to [this code block](https://github.com/apache/arrow/blob/master/r/src/compute-exec.cpp#L396-L406) is outdated. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on issue #15151: Should `GetIOThreadPool()` be accessible from installed headers?

Posted by GitBox <gi...@apache.org>.
westonpace commented on issue #15151:
URL: https://github.com/apache/arrow/issues/15151#issuecomment-1369035221

   Yes, I think it's entirely appropriate to put RecordBatchReader->source node in the C++ code.  On the output side we have collector variants for tables (DeclarationToTable), vector of record batches (DeclarationToBatches) and record batch reader (DeclarationToReader).  We already have source node variants for accepting data from a table (`table_source`) and a vector of record batches (`record_batch_source`).  So I think it would be a good addition to add `record_batch_reader_source`.
   
   However, I'm also not sure why we wouldn't expose the default I/O pool (`arrow::io::internal::GetIOThreadPool`) in the public headers given that we have public methods for getting and setting the size.
   
   As a short term hack you can do:
   
   ```
   #include "arrow/io/type_fwd.h"
   
   arrow::io::IOContext io_context = arrow::io::default_io_context();
   arrow::internal::Executor* io_executor = io_context.executor();
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] vibhatha commented on issue #15151: Should `GetIOThreadPool()` be accessible from installed headers?

Posted by GitBox <gi...@apache.org>.
vibhatha commented on issue #15151:
URL: https://github.com/apache/arrow/issues/15151#issuecomment-1370613697

   @westonpace @paleolimbot I created a draft PR to use a `RecordBatchReader` directly as the data source. I would appreciate your reviews to see if it addresses the problem as expected. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace closed issue #15151: Should `GetIOThreadPool()` be accessible from installed headers?

Posted by GitBox <gi...@apache.org>.
westonpace closed issue #15151: Should `GetIOThreadPool()` be accessible from installed headers?
URL: https://github.com/apache/arrow/issues/15151


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] paleolimbot commented on issue #15151: Should `GetIOThreadPool()` be accessible from installed headers?

Posted by GitBox <gi...@apache.org>.
paleolimbot commented on issue #15151:
URL: https://github.com/apache/arrow/issues/15151#issuecomment-1369082235

   > As a short term hack you can do:
   
   We do this already in the R package for the place where we need to use the IO thread pool to submit jobs...the problem here is that we need a `ThreadPool*` to pass to `MakeReaderGenerator()`.
   
   > I think it would be a good addition to add `record_batch_reader_source`.
   
   That would be my preferred solution...I'd rather not maintain the best way to do that in the R package and it's come up on the mailing list in a context unrelated to the R package as well ( https://lists.apache.org/thread/zo9qq0pntkrt2vnczoxx7hfsl6k233zy ).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] vibhatha commented on issue #15151: Should `GetIOThreadPool()` be accessible from installed headers?

Posted by GitBox <gi...@apache.org>.
vibhatha commented on issue #15151:
URL: https://github.com/apache/arrow/issues/15151#issuecomment-1369362737

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] paleolimbot commented on issue #15151: Should `GetIOThreadPool()` be accessible from installed headers?

Posted by GitBox <gi...@apache.org>.
paleolimbot commented on issue #15151:
URL: https://github.com/apache/arrow/issues/15151#issuecomment-1369864818

   Yes, sorry! The block was this one:
   
   https://github.com/apache/arrow/blob/63b91cc1f7131356537ab9cbb84ed108d6f9102e/r/src/compute-exec.cpp#L459-L468


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org