You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/04/27 14:32:31 UTC
[GitHub] [arrow] pitrou opened a new pull request #10174: ARROW-12554: [C++] Allow duplicates in `SetLookupOptions::value_set`
pitrou opened a new pull request #10174:
URL: https://github.com/apache/arrow/pull/10174
For the `index_in` function, we need to map the memo table indices to indices in the value_set
(they are different in there are duplicates).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #10174: ARROW-12554: [C++] Allow duplicates in `SetLookupOptions::value_set`
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #10174:
URL: https://github.com/apache/arrow/pull/10174#discussion_r621993183
##########
File path: cpp/src/arrow/compute/kernels/scalar_set_lookup.cc
##########
@@ -75,6 +95,9 @@ struct SetLookupState : public KernelState {
using MemoTable = typename HashTraits<Type>::MemoTableType;
MemoTable lookup_table;
+ // When there are duplicates in value_set, the MemoTable indices must
+ // be mapped back to indices in the value_set.
+ std::vector<int32_t> memo_index_to_value_index;
Review comment:
We would some lose performance in exchange of a little space saving. The hash table is much larger, I don't think this is desirable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10174: ARROW-12554: [C++] Allow duplicates in `SetLookupOptions::value_set`
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10174:
URL: https://github.com/apache/arrow/pull/10174#issuecomment-827668622
https://issues.apache.org/jira/browse/ARROW-12554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed pull request #10174: ARROW-12554: [C++] Allow duplicates in `SetLookupOptions::value_set`
Posted by GitBox <gi...@apache.org>.
pitrou closed pull request #10174:
URL: https://github.com/apache/arrow/pull/10174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on a change in pull request #10174: ARROW-12554: [C++] Allow duplicates in `SetLookupOptions::value_set`
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on a change in pull request #10174:
URL: https://github.com/apache/arrow/pull/10174#discussion_r621960840
##########
File path: cpp/src/arrow/compute/kernels/scalar_set_lookup.cc
##########
@@ -75,6 +95,9 @@ struct SetLookupState : public KernelState {
using MemoTable = typename HashTraits<Type>::MemoTableType;
MemoTable lookup_table;
+ // When there are duplicates in value_set, the MemoTable indices must
+ // be mapped back to indices in the value_set.
+ std::vector<int32_t> memo_index_to_value_index;
Review comment:
If there are not many duplicated values (common case?), looks it's possible to compress this table by storing only indices where duplicated values are met.
A quick example. I think we can leave it as follow up task if it works.
```
values a a b a b b c
value_index 0 1 2 3 4 5 6
// memo index to value index mapping
memo_index_to_value_index[0] = 0 --> a
memo_index_to_value_index[1] = 2 --> b
memo_index_to_value_index[2] = 6 --> c
// duplicated value indices in sorted order
duplicated_indices = (1, 3, 4, 5)
// lookup 'a'
memo_index = lookup('a') = 0
0 < duplicated_indices[0]
return 0
// lookup 'b'
memo_index = lookup('b') = 1
1 >= duplicated_indices[0]
memo_index++ // = 2
2 < duplicated_indices[1]
return 2
// lookup 'c'
memo_index = lookup('c') = 2
2 >= duplicated_indices[0]
memo_index++ // = 3
3 >= duplicated_indices[1]
memo_index++ // = 4
4 >= duplicated_indices[2]
memo_index++ // = 5
5 >= duplicated_indices[3]
memo_index++ // = 6
return 6
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org