You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Vishwanatha-HD (via GitHub)" <gi...@apache.org> on 2025/11/21 18:48:50 UTC

[PR] GH-48218 Fix Util & Level Conversion logic to enable Parquet DB suppo… [arrow]

Vishwanatha-HD opened a new pull request, #48219:
URL: https://github.com/apache/arrow/pull/48219

   …rt on s390x
   
   ### Rationale for this change
   
   This PR is intended to enable Parquet DB support on Big-endian (s390x) systems. The fix in this PR fixes the "util & level_conversion" logic.
   
   ### What changes are included in this PR?
   
   The fix includes changes to following file:
   cpp/src/parquet/level_conversion_inc.h
   cpp/src/parquet/test_util.h
   
   ### Are these changes tested?
   
   Yes. The changes are tested on s390x arch to make sure things are working fine. The fix is also tested on x86 arch, to make sure there is no new regression introduced.
   
   ### Are there any user-facing changes?
   
   No.
   
   GitHub main Issue link: https://github.com/apache/arrow/issues/48151


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-48218: [C++][Parquet] Fix Util & Level Conversion logic on big-endian [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #48219:
URL: https://github.com/apache/arrow/pull/48219


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-48218: [C++][Parquet] Fix Util & Level Conversion logic to enable Parquet DB support on s390x [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #48219:
URL: https://github.com/apache/arrow/pull/48219#discussion_r2555536985


##########
cpp/src/parquet/geospatial/util_internal.cc:
##########
@@ -162,7 +162,7 @@ void WKBGeometryBounder::MergeGeometry(::arrow::util::span<const uint8_t> bytes_
 
 void WKBGeometryBounder::MergeGeometryInternal(WKBBuffer* src, bool record_wkb_type) {
   uint8_t endian = src->ReadUInt8();
-#if defined(ARROW_LITTLE_ENDIAN)
+#if ARROW_LITTLE_ENDIAN

Review Comment:
   Nice catch!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-48218: [C++][Parquet] Fix Util & Level Conversion logic on big-endian [arrow]

Posted by "Vishwanatha-HD (via GitHub)" <gi...@apache.org>.
Vishwanatha-HD commented on code in PR #48219:
URL: https://github.com/apache/arrow/pull/48219#discussion_r2564911547


##########
cpp/src/parquet/geospatial/util_internal.cc:
##########
@@ -162,7 +162,7 @@ void WKBGeometryBounder::MergeGeometry(::arrow::util::span<const uint8_t> bytes_
 
 void WKBGeometryBounder::MergeGeometryInternal(WKBBuffer* src, bool record_wkb_type) {
   uint8_t endian = src->ReadUInt8();
-#if defined(ARROW_LITTLE_ENDIAN)
+#if ARROW_LITTLE_ENDIAN

Review Comment:
   Thanks @pitrou 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-48218 Fix Util & Level Conversion logic to enable Parquet DB suppo… [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48219:
URL: https://github.com/apache/arrow/pull/48219#issuecomment-3564227554

   :warning: GitHub issue #48218 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-48218: [C++][Parquet] Fix Util & Level Conversion logic to enable Parquet DB support on s390x [arrow]

Posted by "k8ika0s (via GitHub)" <gi...@apache.org>.
k8ika0s commented on PR #48219:
URL: https://github.com/apache/arrow/pull/48219#issuecomment-3568376195

   @Vishwanatha-HD
   
   Something I’ve bumped into on s390x is that the swap decision sometimes needs to follow the *encoded* byte-order marker more closely than the compile-time `ARROW_LITTLE_ENDIAN` macro. A couple of the downstream readers (especially in geospatial stats) assume “LE on disk” is its own concept, even if the host is BE. So I’m curious how this change behaves when the host is big-endian but the consuming path is trying to normalize everything toward a canonical LE interpretation.
   
   On the level-bitmap side, I’ve occasionally seen the bitmap comparison logic behave differently depending on whether the input originated on a BE machine or came directly from Arrow buffers that were already LE-canonical. Wrapping `GreaterThanBitmap` with a swap might be exactly what’s needed here — I’m mostly wondering how consistently the upstream paths feed this comparison LE vs. host-order data.
   
   Not flagging this as wrong; just passing along some of the odd little behaviors I’ve seen on the BE side of things. Happy to help run this through a few combinations on actual s390x hardware if that would be useful.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-48218: [C++][Parquet] Fix Util & Level Conversion logic on big-endian [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #48219:
URL: https://github.com/apache/arrow/pull/48219#issuecomment-3571178214

   After merging your PR, Conbench analyzed the 2 benchmarking runs that have been run so far on merge-commit 1044022d596cc334eb5662959fbea0120590b623.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/56235372091) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org