You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "caiwanli (via GitHub)" <gi...@apache.org> on 2023/12/05 09:24:06 UTC
[I] How to implement Radix Hash Join for two tables in Arrow? [arrow]
caiwanli opened a new issue, #39071:
URL: https://github.com/apache/arrow/issues/39071
### Describe the usage question you have. Please include as many useful details as possible.
In my join operator, all data will be cached. The cache is defined as follows:
`std::vector<std::shared_ptr<arrow::RecordBatch>> buffer_chunk;`
After caching all the data, I intend to perform Radix partitioning on the buffer_chunk. The function interface is designed as follows:
`void radix_partition(vector<shared_ptr<arrow::RecordBatch>> &input, std::shared_ptr<arrow::RecordBatch> &output, int *histogram);`
After the function execution, the output will be the reordered data, and the "histogram" represents the histogram (used to record the starting position of each partition).
For example, for the data in table 1, dividing the table into 4 partitions results in the data becoming the data in table 2, and simultaneously, the histogram is {0, 3, 7, 9}.
For the implementation of the radix_partition function, there are several issues that need to be addressed:
1. How to efficiently traverse the "input" data?
2. How to efficiently insert data into the "output" while traversing?
Partition calculation method:
`part = tar & （1 << 2）;`
For example,
```
part(57):57 & ((1 << 2) - 1 ) = 1,
part(92):92 & ((1 << 2) - 1 ) = 0
```
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to implement Radix Hash Join for two tables in Arrow? [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #39071: How to implement Radix Hash Join for two tables in Arrow?
URL: https://github.com/apache/arrow/issues/39071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] How to implement Radix Hash Join for two tables in Arrow? [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39071:
URL: https://github.com/apache/arrow/issues/39071#issuecomment-3547060218
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this usage question has evolved into a feature request or docs update, please remove the 'Type: usage' label and add the 'Type: enhancement' label instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org