You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "daniel-adam-tfs (via GitHub)" <gi...@apache.org> on 2026/02/13 18:20:18 UTC
[PR] [Parquet] fix bss and tests on big endian [arrow-go]
daniel-adam-tfs opened a new pull request, #663:
URL: https://github.com/apache/arrow-go/pull/663
### Rationale for this change
To ensure the Arrow and Parquet Go libraries work correctly on big-endian architectures.
### What changes are included in this PR?
Added endianness-aware BYTE_STREAM_SPLIT decoding in the parquet/encoding package.
Fixed tests in the parquet package to handle byte order correctly on big-endian systems.
### Are these changes tested?
Yes, all affected unit tests now pass on both little-endian and big-endian machines. The changes specifically address some of the previously failing tests on big-endian systems.
### Are there any user-facing changes?
No user-facing API changes. The changes are internal and ensure correct behavior on supported architectures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(parquet): bss encoding and tests on big endian systems [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on PR #663:
URL: https://github.com/apache/arrow-go/pull/663#issuecomment-3908028868
@zeroshade I was thinking of maybe adding a CI run on a BE architecture. I'm using docker with QEMU emulation of a s390x machine to run this locally, so I'm sure it's possible in a GitHub action as well. The problem is that there are still couple of dozens tests failing, I've only fixed those in the parquet/encoding. So either we make it run just in the parquet/encoding package for now, or maybe add a job that doesn't run automatically, but can be run manually to see the status.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Parquet] Fix bss encoding and tests on big endian systems [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #663:
URL: https://github.com/apache/arrow-go/pull/663#discussion_r2808025537
##########
parquet/file/file_writer_test.go:
##########
@@ -669,6 +673,13 @@ func NewColumnIndexObject(colIdx metadata.ColumnIndex) (ret ColumnIndexObject) {
}
func simpleEncode[T int32 | int64 | float32 | float64](val T) []byte {
+ if endian.IsBigEndian {
+ buf := bytes.NewBuffer(nil)
+ if err := binary.Write(buf, binary.LittleEndian, val); err != nil {
+ panic(err)
+ }
+ return buf.Bytes()
+ }
Review Comment:
there is a `ToLE` function in `internal/utils/` that might be better for this. For example:
```go
val = utils.ToLE(val)
return unsafe.Slice((*byte)(unsafe.Pointer(&val)), unsafe.Sizeof(val))
```
This would work because on LittleEndian systems, `ToLE(val)` is just defined as `return val`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(parquet): bss encoding and tests on big endian systems [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on code in PR #663:
URL: https://github.com/apache/arrow-go/pull/663#discussion_r2812039631
##########
parquet/internal/encoding/byte_stream_split_big_endian.go:
##########
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build armbe || arm64be || m68k || mips || mips64 || mips64p32 || ppc || ppc64 || s390 || s390x || shbe || sparc || sparc64
Review Comment:
I've copied these from https://golang.bg/src/encoding/binary/native_endian_big.go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(parquet): bss encoding and tests on big endian systems [arrow-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #663:
URL: https://github.com/apache/arrow-go/pull/663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(parquet): bss encoding and tests on big endian systems [arrow-go]
Posted by "daniel-adam-tfs (via GitHub)" <gi...@apache.org>.
daniel-adam-tfs commented on code in PR #663:
URL: https://github.com/apache/arrow-go/pull/663#discussion_r2812001186
##########
parquet/file/file_writer_test.go:
##########
@@ -669,6 +673,13 @@ func NewColumnIndexObject(colIdx metadata.ColumnIndex) (ret ColumnIndexObject) {
}
func simpleEncode[T int32 | int64 | float32 | float64](val T) []byte {
+ if endian.IsBigEndian {
+ buf := bytes.NewBuffer(nil)
+ if err := binary.Write(buf, binary.LittleEndian, val); err != nil {
+ panic(err)
+ }
+ return buf.Bytes()
+ }
Review Comment:
Yeah, that works and it is simpler. Let me fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org