You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "thisisnic (via GitHub)" <gi...@apache.org> on 2026/07/09 15:32:18 UTC
[PR] GH-50378: [R] Reading a parquet with a Float16 column yields incorrect value [arrow]
thisisnic opened a new pull request, #50451:
URL: https://github.com/apache/arrow/pull/50451
### Rationale for this change
Wrong values reading Parquet files with float16s in them as we were returning the raw bytes not the float value
### What changes are included in this PR?
Return the float value. Also enabling float16 in a lot of our tests.
### Are these changes tested?
Aye
### Are there any user-facing changes?
Nah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50378: [R] Reading a parquet with a Float16 column yields incorrect value [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50451:
URL: https://github.com/apache/arrow/pull/50451#discussion_r3615033658
##########
r/src/r_to_arrow.cpp:
##########
@@ -390,12 +391,12 @@ struct RConvert {
return static_cast<float>(from);
}
- // ---- convert to half float: not implemented
+ // ---- convert to half float
template <typename Type, typename From>
static enable_if_t<std::is_same<Type, const HalfFloatType>::value,
Result<typename Type::c_type>>
Convert(Type*, From from) {
- return Status::Invalid("Cannot convert to Half Float");
Review Comment:
My point is that float32s and float64s are weird too, so it seems a bit weird to special-case float16 in the name of shielding the user from the weirdness :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50378: [R] Reading a parquet with a Float16 column yields incorrect value [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on code in PR #50451:
URL: https://github.com/apache/arrow/pull/50451#discussion_r3568702532
##########
r/src/r_to_arrow.cpp:
##########
@@ -390,12 +391,12 @@ struct RConvert {
return static_cast<float>(from);
}
- // ---- convert to half float: not implemented
+ // ---- convert to half float
template <typename Type, typename From>
static enable_if_t<std::is_same<Type, const HalfFloatType>::value,
Result<typename Type::c_type>>
Convert(Type*, From from) {
- return Status::Invalid("Cannot convert to Half Float");
Review Comment:
Sure, I get how floats work, but we don't expose all of the C++ API in R, and I'm weighing up where this sits in terms of balancing functionality for advanced users without introducing complexity for less advanced users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50378: [R] Reading a parquet with a Float16 column yields incorrect value [arrow]
Posted by "jonkeane (via GitHub)" <gi...@apache.org>.
jonkeane commented on code in PR #50451:
URL: https://github.com/apache/arrow/pull/50451#discussion_r3558817323
##########
r/src/r_to_arrow.cpp:
##########
@@ -390,12 +391,12 @@ struct RConvert {
return static_cast<float>(from);
}
- // ---- convert to half float: not implemented
+ // ---- convert to half float
template <typename Type, typename From>
static enable_if_t<std::is_same<Type, const HalfFloatType>::value,
Result<typename Type::c_type>>
Convert(Type*, From from) {
- return Status::Invalid("Cannot convert to Half Float");
Review Comment:
I'm surprised we didn't get this error when someone tried this. Any ideas why this didn't bonk? It's not super important for us here since we have now implemented this, but I am worried there might be other things that we have that are supposed to bonk like this but actually aren't 😬
##########
r/tests/testthat/test-chunked-array.R:
##########
Review Comment:
Do we need a line like this to actually test the roundtripping?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50378: [R] Reading a parquet with a Float16 column yields incorrect value [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on code in PR #50451:
URL: https://github.com/apache/arrow/pull/50451#discussion_r3559277635
##########
r/src/r_to_arrow.cpp:
##########
@@ -390,12 +391,12 @@ struct RConvert {
return static_cast<float>(from);
}
- // ---- convert to half float: not implemented
+ // ---- convert to half float
template <typename Type, typename From>
static enable_if_t<std::is_same<Type, const HalfFloatType>::value,
Result<typename Type::c_type>>
Convert(Type*, From from) {
- return Status::Invalid("Cannot convert to Half Float");
Review Comment:
That bit there is enabling the R to Arrow direction, but the original issue was hitting the Arrow to R direction. My assumption is that we chose not to implement it because it's not really useful for most R users to have it? Like float16s are weird because you can't represent as many numbers, e.g.
```r
> arrow_array(c(1.1, 2.2, 3.3), float16())
Array
<halffloat>
[
1.099609375,
2.19921875,
3.30078125
]
```
And actually, they don't roundtrip nicely at all
```r
> arrow_array(c(1.1, 2.2, 3.3), float16())$as_vector()
[1] 1.099609 2.199219 3.300781
```
Though you could say 1.1 was never a float16 in the first place. Maybe I should just ditch the R to Arrow stuff; it felt like making a more complete PR, but we might just be handing users a footgun, as I can image finding it easy to go "this will take up less storage, let's use it".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50378: [R] Reading a parquet with a Float16 column yields incorrect value [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50451:
URL: https://github.com/apache/arrow/pull/50451#discussion_r3560041862
##########
r/src/r_to_arrow.cpp:
##########
@@ -390,12 +391,12 @@ struct RConvert {
return static_cast<float>(from);
}
- // ---- convert to half float: not implemented
+ // ---- convert to half float
template <typename Type, typename From>
static enable_if_t<std::is_same<Type, const HalfFloatType>::value,
Result<typename Type::c_type>>
Convert(Type*, From from) {
- return Status::Invalid("Cannot convert to Half Float");
Review Comment:
1.1 was never a float64 either :)
```python
>>> 1.1 - 1
0.10000000000000009
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50378: [R] Reading a parquet with a Float16 column yields incorrect value [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50451:
URL: https://github.com/apache/arrow/pull/50451#discussion_r3552896039
##########
r/tests/testthat/test-dplyr-funcs-type.R:
##########
@@ -31,7 +31,7 @@ test_that("explicit type conversions with cast()", {
int_types <- c(int8(), int16(), int32(), int64())
uint_types <- c(uint8(), uint16(), uint32(), uint64())
- float_types <- c(float32(), float64())
+ float_types <- c(float16(), float32(), float64())
Review Comment:
Adding `float16()` to `float_types` here expands type-coverage but still doesn't assert that Float16 values roundtrip back to R correctly (the GH-50378 regression was about decoding values, not just preserving the schema type). Consider adding a small regression check that casts to `float16()` and then `collect()`s to ensure the resulting R value matches the input (and is not the raw uint16 bits).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50378: [R] Reading a parquet with a Float16 column yields incorrect value [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on code in PR #50451:
URL: https://github.com/apache/arrow/pull/50451#discussion_r3804057793
##########
r/tests/testthat/test-chunked-array.R:
##########
Review Comment:
Added a new test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50378: [R] Reading a parquet with a Float16 column yields incorrect value [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic merged PR #50451:
URL: https://github.com/apache/arrow/pull/50451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org