You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "chelseajonesr (via GitHub)" <gi...@apache.org> on 2023/09/13 22:31:26 UTC
[GitHub] [arrow] chelseajonesr opened a new issue, #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
chelseajonesr opened a new issue, #37712:
URL: https://github.com/apache/arrow/issues/37712
### Describe the bug, including details regarding any error messages, version, and platform.
When reading Parquet files containing a nested field that is null for all records, it panics when using go 1.21.1 but not when using go 1.20.8.
This is a small reproducible case:
https://github.com/chelseajonesr/rfarrow/blob/f7bef4585eeb4d7582014bc54b0fa7092651f774/panic_test.go#L13
The panic:
```
panic: runtime error: slice bounds out of range [:5] with capacity 0
goroutine 15 [running]:
github.com/apache/arrow/go/v13/parquet/internal/encoding.(*PlainByteArrayDecoder).DecodeSpaced(0x0?, {0x0?, 0x14000307ce8?, 0x103398720?}, 0x103d2ff20?, {0x14000140980?, 0x100307c01?, 0x2?}, 0x400010000001f?)
/Users/chelseajones/go/pkg/mod/github.com/apache/arrow/go/v13@v13.0.0/parquet/internal/encoding/byte_array_decoder.go:83 +0x130
github.com/apache/arrow/go/v13/parquet/file.(*byteArrayRecordReader).ReadValuesSpaced(0x1400017eb00, 0x0, 0x20?)
/Users/chelseajones/go/pkg/mod/github.com/apache/arrow/go/v13@v13.0.0/parquet/file/record_reader.go:841 +0x134
github.com/apache/arrow/go/v13/parquet/file.(*recordReader).ReadRecordData(0x14000103260, 0x1)
/Users/chelseajones/go/pkg/mod/github.com/apache/arrow/go/v13@v13.0.0/parquet/file/record_reader.go:548 +0x288
github.com/apache/arrow/go/v13/parquet/file.(*recordReader).ReadRecords(0x14000103260, 0x1)
/Users/chelseajones/go/pkg/mod/github.com/apache/arrow/go/v13@v13.0.0/parquet/file/record_reader.go:632 +0x32c
github.com/apache/arrow/go/v13/parquet/pqarrow.(*leafReader).LoadBatch(0x140001032c0, 0x1)
/Users/chelseajones/go/pkg/mod/github.com/apache/arrow/go/v13@v13.0.0/parquet/pqarrow/column_readers.go:103 +0xd8
github.com/apache/arrow/go/v13/parquet/pqarrow.(*structReader).LoadBatch.func1()
/Users/chelseajones/go/pkg/mod/github.com/apache/arrow/go/v13@v13.0.0/parquet/pqarrow/column_readers.go:241 +0x30
golang.org/x/sync/errgroup.(*Group).Go.func1()
/Users/chelseajones/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 13
/Users/chelseajones/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x98
```
### Component(s)
Go, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37712:
URL: https://github.com/apache/arrow/issues/37712#issuecomment-1719752532
@chelseajonesr Using v14 (the current main) I am able to correctly read the file from the reproducible case you linked to. I'm guessing this was likely related to the null handling bug in other decoding types or the fix that was made to the bitmap handler. I haven't been able to identify specifically *which* change fixed the problem, but it does appear that this is fixed in the latest version. Can you try pulling down v14 and confirming this for yourself, just to ensure I didn't miss something?
The v14 release should happen around mid-late October which would contain this fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] chelseajonesr commented on issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
Posted by "chelseajonesr (via GitHub)" <gi...@apache.org>.
chelseajonesr commented on issue #37712:
URL: https://github.com/apache/arrow/issues/37712#issuecomment-1723930929
I stepped through with go 1.20.8 and then 1.21.1:
With 1.20.8, the call `funclist.extractBits(bitmap, selectBitmap)` in `ExtractBits()` (line 38 of bmi.go) calls `extractBitsGo()` (line 242 of bmi.go). Both `bitmap` and `selectBitmap` are 0; the call returns 0.
With 1.21.1, the call `funclist.extractBits(bitmap, selectBitmap)` in `ExtractBits()` (line 38 of bmi.go) calls `extractBitsNEON` (line 29 of bitmap_neon_arm64.go). Again, `bitmap` and `selectBitmap` are 0, but this call returns 1. Then the nullCount in ReadRecordData() is -1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] chelseajonesr commented on issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
Posted by "chelseajonesr (via GitHub)" <gi...@apache.org>.
chelseajonesr commented on issue #37712:
URL: https://github.com/apache/arrow/issues/37712#issuecomment-1721863745
I just asked a coworker to give it a try. On his MacOS/ARM64 machine, it panics. On an older MacOS/x86, it did not panic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #37712:
URL: https://github.com/apache/arrow/issues/37712#issuecomment-1721866843
Maybe it's related to https://github.com/apache/arrow/pull/37112 ? Since RLE/bit-pack for rep-def levels will depend on BitWriter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37712:
URL: https://github.com/apache/arrow/issues/37712#issuecomment-1723937289
So it's definitely an issue with the arm NEON code, which is what I was thinking (still getting myself access to an ARM machine) though I don't know why it doesn't use the NEON implementation in go1.20.8 for you and then does use it in 1.21.1.
For now you should be able to avoid the crash by using the `noasm` tag which will force it to use the pure go implementation instead of the ARM neon assembly implementation. I'll see if I can track down where the issue is in the arm neon code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] chelseajonesr commented on issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
Posted by "chelseajonesr (via GitHub)" <gi...@apache.org>.
chelseajonesr commented on issue #37712:
URL: https://github.com/apache/arrow/issues/37712#issuecomment-1719805064
@zeroshade Thanks for the quick response. I have a fork of arrow that I've just synched to the head of main; I copied the panic test there and I'm still seeing the same behaviour.
https://github.com/chelseajonesr/arrow/blob/panic-using-go-1.21/go/parquet/panic_test.go
Go 1.20.8:
```
parquet % go test ./panic_test.go
ok command-line-arguments 0.224s
```
And then Go 1.21.1:
```
parquet % ~/go/bin/go1.21.1 test ./panic_test.go
panic: runtime error: slice bounds out of range [:5] with capacity 0
goroutine 14 [running]:
github.com/apache/arrow/go/v14/parquet/internal/encoding.(*PlainByteArrayDecoder).DecodeSpaced(0x0?, {0x0?, 0x1400030bce8?, 0x1031e5200?}, 0x103b86ab8?, {0x140000ce900?, 0x10030bc01?, 0x2?}, 0x400010000001f?)
/Users/chelseajones/src/arrow-fork/go/parquet/internal/encoding/byte_array_decoder.go:83 +0x130
github.com/apache/arrow/go/v14/parquet/file.(*byteArrayRecordReader).ReadValuesSpaced(0x140000feb00, 0x0, 0x20?)
/Users/chelseajones/src/arrow-fork/go/parquet/file/record_reader.go:841 +0x134
github.com/apache/arrow/go/v14/parquet/file.(*recordReader).ReadRecordData(0x14000095260, 0x1)
/Users/chelseajones/src/arrow-fork/go/parquet/file/record_reader.go:548 +0x288
github.com/apache/arrow/go/v14/parquet/file.(*recordReader).ReadRecords(0x14000095260, 0x1)
/Users/chelseajones/src/arrow-fork/go/parquet/file/record_reader.go:632 +0x32c
github.com/apache/arrow/go/v14/parquet/pqarrow.(*leafReader).LoadBatch(0x140000952c0, 0x1)
/Users/chelseajones/src/arrow-fork/go/parquet/pqarrow/column_readers.go:104 +0xd8
github.com/apache/arrow/go/v14/parquet/pqarrow.(*structReader).LoadBatch.func1()
/Users/chelseajones/src/arrow-fork/go/parquet/pqarrow/column_readers.go:242 +0x30
golang.org/x/sync/errgroup.(*Group).Go.func1()
/Users/chelseajones/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 18
/Users/chelseajones/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0x98
FAIL command-line-arguments 0.315s
FAIL
```
If this is working for you... I wonder if there's some sort of configuration issue I'm running into.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] chelseajonesr commented on issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
Posted by "chelseajonesr (via GitHub)" <gi...@apache.org>.
chelseajonesr commented on issue #37712:
URL: https://github.com/apache/arrow/issues/37712#issuecomment-1723955055
Great, thanks for the workaround!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1 [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
URL: https://github.com/apache/arrow/issues/37712
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37712:
URL: https://github.com/apache/arrow/issues/37712#issuecomment-1723588382
@mapleFU I was thinking that too, but I don't know how that would cause a difference in whether or not it panics. Though it looks like the difference between the panics seems to definitely be related to ARM. Possibly the NEON code? Gonna get am ARM machine to try to debug and see what I can figure out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37712:
URL: https://github.com/apache/arrow/issues/37712#issuecomment-1721839282
Hmm, so running with `go1.21.1` myself locally (amd64 linux) i'm not able to reproduce this failure. I'll see if i can try to get set up in an arm64 / macos env and test there to try to replicate the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #37712: [Go][Parquet] Reading a parquet file with a nested field that is null for all records panics when using Go 1.21.1
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #37712:
URL: https://github.com/apache/arrow/issues/37712#issuecomment-1724499970
@chelseajonesr I've figured out a solution (and why it was only failing with `go1.21`, you can see the fix on the linked PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org