You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "ella-chao (via GitHub)" <gi...@apache.org> on 2023/11/29 17:42:51 UTC
[I] [Go] Allow access to the underlying `MemoTable` of a dictionary builder [arrow]
ella-chao opened a new issue, #38988:
URL: https://github.com/apache/arrow/issues/38988
### Describe the enhancement requested
I have a case where knowing the size of the dictionary as values get appended to the dictionary builder will be useful. Specifically, I am indexing data where the number of unique values is unknown. As the number of unique values is more likely to be relatively small in this case, a `BinaryDictionaryBuilder` is used and only when it is detected that the dictionary will be too big do I fall back to a `LargeStringBuilder`.
The issue is that there is no easy way to figure out the size of the dictionary in a `BinaryDictionaryBuilder` today. As a workaround, after each `AppendString` to the `BinaryDictionaryBuilder` I do the following
```
lastDictIndex := bldrDictString.(*arrowarray.BinaryDictionaryBuilder).GetValueIndex(i)
if lastDictIndex+1 > cardinality {
cardinality = lastDictIndex + 1
}
```
where `i` is the index of the value appended.
It would be more convenient and potentially less costly if the `MemoTable` or even just the size of the dictionary is exposed. Do you think this is something that you will be open to? I will be happy to open a PR if so.
### Component(s)
Go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] Allow access to the underlying `MemoTable` of a dictionary builder [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #38988:
URL: https://github.com/apache/arrow/issues/38988#issuecomment-1847706784
Hey @ella-chao sorry for the delay in response here. I think it's definitely a good idea to expose the current size of the dictionary from the builder. I'd be happy to review a PR if you file it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] Allow access to the underlying `MemoTable` of a dictionary builder [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #38988: [Go] Allow access to the underlying `MemoTable` of a dictionary builder
URL: https://github.com/apache/arrow/issues/38988
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] Allow access to the underlying `MemoTable` of a dictionary builder [arrow]
Posted by "ella-chao (via GitHub)" <gi...@apache.org>.
ella-chao commented on issue #38988:
URL: https://github.com/apache/arrow/issues/38988#issuecomment-1882004522
Hello, no worries and thanks for getting back to me. Apologies for the late response on my end as well. Per your recommendation I've opened https://github.com/apache/arrow/pull/39521 that exposes the dictionary size only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org