You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/22 22:18:55 UTC
[GitHub] [arrow] anjakefala opened a new pull request, #13947: ARROW-17464: [C++] Store/Read float16 values in Parquet as FixedSizeBinary
anjakefala opened a new pull request, #13947:
URL: https://github.com/apache/arrow/pull/13947
Half-float values are currently not supported in Parquet:
https://issues.apache.org/jira/browse/PARQUET-1647
This is a draft PR! The tests as they are currently not passing. I am a pretty new committer, and opened this PR to more easily ask for help understanding what various parts of the code are doing.
A pre-requisite for any merging is to submit to the Parquet mailing list a proposal to have this standardised in Parquet itself. I will link to my post here once I send it.
This PR was inspired by: https://github.com/apache/arrow/pull/12449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13947: ARROW-17464: [C++] Store/Read float16 values in Parquet as FixedSizeBinary
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13947:
URL: https://github.com/apache/arrow/pull/13947#issuecomment-1223222728
https://issues.apache.org/jira/browse/ARROW-17464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #13947: ARROW-17464: [C++] Store/Read float16 values in Parquet as FixedSizeBinary
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13947:
URL: https://github.com/apache/arrow/pull/13947#issuecomment-1239793505
I'm not sure it's a good idea to do this before the HALF logical type is standardized in Parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anjakefala commented on pull request #13947: ARROW-17464: [C++] Store/Read float16 values in Parquet as FixedSizeBinary
Posted by GitBox <gi...@apache.org>.
anjakefala commented on PR #13947:
URL: https://github.com/apache/arrow/pull/13947#issuecomment-1233313868
(Force push was needed to update my fork with updates from `apache/arrow` repository. No new changes.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anjakefala commented on pull request #13947: ARROW-17464: [C++] Store/Read float16 values in Parquet as FixedSizeBinary
Posted by GitBox <gi...@apache.org>.
anjakefala commented on PR #13947:
URL: https://github.com/apache/arrow/pull/13947#issuecomment-1238582359
If you run the `TestHalfFloatParquetIO` test, currently you get an `std::bad_cast` stacktrace:
Result of stacktrace:
```
/home/anja/git/arrow/cpp/src/parquet/arrow/arrow_reader_writer_test.cc:664: Failure
Expected: for (::arrow::Status _st = ::arrow::internal::GenericToStatus((WriteTable(*table, ::arrow::default_memory_pool(), this->sink_, values->length(), default_writer_properties(), arrow_properties))); !_st.ok();) return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/anja/git/arrow/cpp/src/parquet/arrow/arrow_reader_writer_test.cc", 664, "Failed") = ::testing::Message() << "'" "WriteTable(*table, ::arrow::default_memory_pool(), this->sink_, values->length(), default_writer_properties(), arrow_properties)" "' failed with " << _st.ToString() doesn't throw an exception.
Actual: it throws std::bad_cast with description "std::bad_cast".
/home/anja/git/arrow/cpp/src/parquet/arrow/arrow_reader_writer_test.cc:664: Failure
Expected: for (::arrow::Status _st = ::arrow::internal::GenericToStatus((WriteTable(*table, ::arrow::default_memory_pool(), this->sink_, values->length(), default_writer_properties(), arrow_properties))); !_st.ok();) return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/anja/git/arrow/cpp/src/parquet/arrow/arrow_reader_writer_test.cc", 664, "Failed") = ::testing::Message() << "'" "WriteTable(*table, ::arrow::default_memory_pool(), this->sink_, values->length(), default_writer_properties(), arrow_properties)" "' failed with " << _st.ToString() doesn't throw an exception.
Actual: it throws std::bad_cast with description "std::bad_cast".
```
I managed to narrow this stacktrace down to this call:
```
WRITE_SERIALIZE_CASE(HALF_FLOAT, FixedSizeBinaryType, FLBAType)
```
in `src/cpp/parquet/column_writer.cc`
This will then call:
- `WriteArrowSerialize<FLBAType, FixedSizeBinaryType>()`
With the `std::bad_cast` being thrown by -> `RETURN_NOT_OK(functor.Serialize(checked_cast<const ArrayType&>(array), ctx, buffer)); (l1569)
- it uses ArrayType -> typename ::arrow::TypeTraits<FixedSizeBinaryType>::ArrayType;
- it uses SerializeFunctor<FLBAType, FixedSizeBinaryType>
I am going to continue trying to understand this, but I am outside my comfort zone, and really welcome help! In particular, my test might need adjustment, and perhaps my `ParquetType` and `ArrowType` should be changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #13947: ARROW-17464: [C++] Store/Read float16 values in Parquet as FixedSizeBinary
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on PR #13947:
URL: https://github.com/apache/arrow/pull/13947#issuecomment-1239207069
My quick guess is that we can't cast a HalfFloatArray to a FixedSizeBinaryArray, and this is causing the crash. For example another case that takes this path in the column writer is Decimal128Array, but that array extends FixedSizeBinaryArray, and thus can be cast to that type.
While it's possible to implement such a cast operator, I am not sure that's something we typically do in our codebase for arrays. I suppose somewhere in the write path, we should explicitly convert the float array to a binary array (it might be this conversion still needs to be implemented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #13947: ARROW-17464: [C++] Store/Read float16 values in Parquet as FixedSizeBinary
Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #13947:
URL: https://github.com/apache/arrow/pull/13947#issuecomment-1239242349
Yes, definitely, you cannot interpret one type of array like this as another.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] rok commented on a diff in pull request #13947: ARROW-17464: [C++] Store/Read float16 values in Parquet as FixedSizeBinary
Posted by GitBox <gi...@apache.org>.
rok commented on code in PR #13947:
URL: https://github.com/apache/arrow/pull/13947#discussion_r965195239
##########
cpp/src/parquet/column_writer.cc:
##########
@@ -2050,6 +2050,7 @@ Status TypedColumnWriterImpl<FLBAType>::WriteArrowDense(
WRITE_SERIALIZE_CASE(FIXED_SIZE_BINARY, FixedSizeBinaryType, FLBAType)
WRITE_SERIALIZE_CASE(DECIMAL128, Decimal128Type, FLBAType)
WRITE_SERIALIZE_CASE(DECIMAL256, Decimal256Type, FLBAType)
+ WRITE_SERIALIZE_CASE(HALF_FLOAT, FixedSizeBinaryType, FLBAType)
Review Comment:
This would resolve the segfault but doesn't really feel elegant.
```suggestion
case ::arrow::Type::HALF_FLOAT: {
auto array_data = array.data();
const auto& arr = ::arrow::FixedSizeBinaryArray(
::arrow::fixed_size_binary(2), array.length(), array_data->buffers[1],
array_data->buffers[0], array.null_count(), array.offset());
return WriteArrowSerialize<FLBAType, ::arrow::FixedSizeBinaryType>(
arr, num_levels, def_levels, rep_levels, ctx, this, maybe_parent_nulls);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] anjakefala closed pull request #13947: ARROW-17464: [C++] Store/Read float16 values in Parquet as FixedSizeBinary
Posted by GitBox <gi...@apache.org>.
anjakefala closed pull request #13947: ARROW-17464: [C++] Store/Read float16 values in Parquet as FixedSizeBinary
URL: https://github.com/apache/arrow/pull/13947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org