You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "stenlarsson (via GitHub)" <gi...@apache.org> on 2024/04/24 09:38:58 UTC

[I] [Ruby] Allow disabling threads for execution plans [arrow]

stenlarsson opened a new issue, #41364:
URL: https://github.com/apache/arrow/issues/41364

   ### Describe the enhancement requested
   
   The aggregators `first` and `last` are unusable in Ruby because they don't work when the execution plan is executed using threads:
   
   ```ruby
   require 'arrow'
   
   table = Arrow::Table.new(
     'foo' => [1, 1],
     'bar' => ['first', 'second'],
   )
   pp table.group('foo').aggregate('first(bar)')
   ```
   
   Executing this script results in:
   
   ```
   [execute-plan][build-node]: NotImplemented: Using ordered aggregator in multiple threaded execution is not supported (Arrow::Error::NotImplemented)
   ```
   
   This seems like the same issue that was fixed for Python in https://github.com/apache/arrow/pull/36768, and similar solution is needed for Ruby.
   
   ### Component(s)
   
   Ruby


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Ruby] Allow disabling threads for execution plans [arrow]

Posted by "stenlarsson (via GitHub)" <gi...@apache.org>.
stenlarsson commented on issue #41364:
URL: https://github.com/apache/arrow/issues/41364#issuecomment-2077001892

   Wouldn't that mean we have to implement bindings for the whole "declaration" API?
   
   I don't fully understand the code, but it seems that when `use_threads` is false it creates a thread pool with only one thread: https://github.com/apache/arrow/blob/de3cdc00c21fd3e9d8d763099591f23720ca8d1f/cpp/src/arrow/acero/exec_plan.cc#L769-L776
   
   The size of the thread pool is then checked by the GroupByNode: https://github.com/apache/arrow/blob/32885c91a1c94f1c8530815f4d4b8fc068859de8/cpp/src/arrow/acero/groupby_aggregate_node.cc#L189
   
   I think if we were able to create an `ExecuteContext` with such a thread pool in Ruby, and then pass that to the constructor of `ExecutePlan` we should be good, right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Ruby] Allow disabling threads for execution plans [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #41364:
URL: https://github.com/apache/arrow/issues/41364#issuecomment-3547522902

   This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Ruby] Allow disabling threads for execution plans [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #41364:
URL: https://github.com/apache/arrow/issues/41364#issuecomment-2076064121

   It makes sense.
   
   We need to implement the bindings of `arrow::acero::QueryOptions` bindings as the first for this:
   
   https://github.com/apache/arrow/blob/d7a5777bb3626d4f448ae4c525a03c8ec6bda5dc/cpp/src/arrow/acero/exec_plan.h#L509-L605
   
   We should create a separated issue for this to work on this step by step.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Ruby] Allow disabling threads for execution plans [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #41364:
URL: https://github.com/apache/arrow/issues/41364#issuecomment-2078212169

   Ah, `ExecPlan` doesn't see `arrow::acero::QueryOptions::use_threads`...
   
   Our options:
   
   1. Add support for ordered aggregation in parallel in C++
   
      #34311 is the PR that implements ordered aggregation.
   
   2. Add support for the declaration API 
   
   3. Add `Arrow::ThreadPool` and use it with `Arrow::ExecutePlan`
   
       We also need to improve `Arrow::ExecuteContext` and so on.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Ruby] Allow disabling threads for execution plans [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #41364:
URL: https://github.com/apache/arrow/issues/41364#issuecomment-3662441284

   Issue resolved by pull request 48462
   https://github.com/apache/arrow/pull/48462


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Ruby] Allow disabling threads for execution plans [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed issue #41364: [Ruby] Allow disabling threads for execution plans
URL: https://github.com/apache/arrow/issues/41364


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Ruby] Allow disabling threads for execution plans [arrow]

Posted by "stenlarsson (via GitHub)" <gi...@apache.org>.
stenlarsson commented on issue #41364:
URL: https://github.com/apache/arrow/issues/41364#issuecomment-3614344997

   I will try to implement this myself.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org