You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "metalmatze (via GitHub)" <gi...@apache.org> on 2023/06/06 16:58:32 UTC
[GitHub] [arrow] metalmatze opened a new pull request, #35956: MINOR: [Go] Add IsNull method to dictionary builder
metalmatze opened a new pull request, #35956:
URL: https://github.com/apache/arrow/pull/35956
Given that this function is super similar to the previously added `GetValueIndex` I've decided to not open an Issue first.
### Rationale for this change
This is helpful to check if previously appended rows have been null or not.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
The changes have been tested in our unit tests building on top of this. I'm happy to follow up with a unit test if deemed necessary.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
The addition of `IsNull(i int) bool` to the `BinaryDictionaryBuilder`.
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #35956: MINOR: [Go] Add IsNull method to dictionary builder
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #35956:
URL: https://github.com/apache/arrow/pull/35956#issuecomment-1587896296
Benchmark runs are scheduled for baseline = ae655c5ccb8d4bec1acd0f6d50855a6dea1590c1 and contender = a4d241c2f7da1ae11a309300a85cc25bc9ec107e. a4d241c2f7da1ae11a309300a85cc25bc9ec107e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/bc3b2b259d7a4be5aeb93256a161ec55...9d136689d4d943d2b30689103b1da442/)
[Finished :arrow_down:1.36% :arrow_up:0.03%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/5832792f543b469b945f38ea52eb408d...fe595a799770475da914b49259ee6363/)
[Finished :arrow_down:0.33% :arrow_up:0.65%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/76780b49d74b45f9ae18e2b5981de5e9...16e992c1b4bf43149a101260e42f2725/)
[Finished :arrow_down:0.95% :arrow_up:0.54%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/4d47e5612317432b9f0d486f1c92eb2f...375d29f0990548338bb669fd31430772/)
Buildkite builds:
[Finished] [`a4d241c2` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/3024)
[Finished] [`a4d241c2` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/3060)
[Finished] [`a4d241c2` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/3025)
[Finished] [`a4d241c2` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/3050)
[Finished] [`ae655c5c` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/3023)
[Finished] [`ae655c5c` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/3059)
[Finished] [`ae655c5c` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/3024)
[Finished] [`ae655c5c` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/3049)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on pull request #35956: MINOR: [Go] Add IsNull method to dictionary builder
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #35956:
URL: https://github.com/apache/arrow/pull/35956#issuecomment-1587603173
Thanks much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] metalmatze commented on pull request #35956: MINOR: [Go] Add IsNull method to dictionary builder
Posted by "metalmatze (via GitHub)" <gi...@apache.org>.
metalmatze commented on PR #35956:
URL: https://github.com/apache/arrow/pull/35956#issuecomment-1587545483
Pushed a basic test to read back the nulls from the generic builder type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on a diff in pull request #35956: MINOR: [Go] Add IsNull method to dictionary builder
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #35956:
URL: https://github.com/apache/arrow/pull/35956#discussion_r1220246993
##########
go/arrow/array/dictionary.go:
##########
@@ -1293,6 +1293,29 @@ func (b *BinaryDictionaryBuilder) InsertStringDictValues(arr *String) (err error
return
}
+func (b *BinaryDictionaryBuilder) IsNull(i int) bool {
+ switch b := b.idxBuilder.Builder.(type) {
+ case *Uint8Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Int8Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Uint16Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Int16Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Uint32Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Int32Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Uint64Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Int64Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ default:
+ return false
+ }
+}
Review Comment:
If you just add `IsNull` to the underlying `builder` struct in `builder.go` then *all* the builders will get it for free as they all embed the base builder. It also means you wouldn't need this switch here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] metalmatze commented on a diff in pull request #35956: MINOR: [Go] Add IsNull method to dictionary builder
Posted by "metalmatze (via GitHub)" <gi...@apache.org>.
metalmatze commented on code in PR #35956:
URL: https://github.com/apache/arrow/pull/35956#discussion_r1221198689
##########
go/arrow/array/dictionary.go:
##########
@@ -1293,6 +1293,29 @@ func (b *BinaryDictionaryBuilder) InsertStringDictValues(arr *String) (err error
return
}
+func (b *BinaryDictionaryBuilder) IsNull(i int) bool {
+ switch b := b.idxBuilder.Builder.(type) {
+ case *Uint8Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Int8Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Uint16Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Int16Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Uint32Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Int32Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Uint64Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ case *Int64Builder:
+ return b.builder.nullBitmap.Len() != 0 && bitutil.BitIsNotSet(b.builder.nullBitmap.Bytes(), i)
+ default:
+ return false
+ }
+}
Review Comment:
That's cool! Let me tackle that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade merged pull request #35956: MINOR: [Go] Add IsNull method to dictionary builder
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #35956:
URL: https://github.com/apache/arrow/pull/35956
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org