You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Illyrix (via GitHub)" <gi...@apache.org> on 2023/08/10 07:24:42 UTC
[GitHub] [arrow] Illyrix opened a new issue, #37102: [Go] [Parquet] panic when writing particular dataset encoded by DeltaBitPacked
Illyrix opened a new issue, #37102:
URL: https://github.com/apache/arrow/issues/37102
### Describe the bug, including details regarding any error messages, version, and platform.
version: v12.0.1
here is the testcase which cause panic:
`parquet/internal/encoding/encoding_test.go`
```go
func TestWriteDeltaBitPackedInt64(t *testing.T) {
column := schema.NewColumn(schema.NewInt64Node("int64", parquet.Repetitions.Required, -1), 0, 0)
tests := []struct {
name string
toencode []int64
expected []byte
}{
{"panic data", []int64{
0, 3000000000000000000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 3000000000000000000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 3000000000000000000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 3000000000000000000, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
}, []byte{0, 0, 0, 0, 0}}, // ignore expected bytes
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
enc := encoding.NewEncoder(parquet.Types.Int64, parquet.Encodings.DeltaBinaryPacked, false, column, memory.DefaultAllocator)
enc.(encoding.Int64Encoder).Put(tt.toencode)
buf, _ := enc.FlushValues() // <------------- panic on this line
defer buf.Release()
assert.Equal(t, tt.expected, buf.Bytes())
dec := encoding.NewDecoder(parquet.Types.Int64, parquet.Encodings.DeltaBinaryPacked, column, memory.DefaultAllocator)
dec.(encoding.Int64Decoder).SetData(len(tt.toencode), buf.Bytes())
out := make([]int64, len(tt.toencode))
dec.(encoding.Int64Decoder).Decode(out)
assert.Equal(t, tt.toencode, out)
})
}
// other subtests...
}
```
### what I expect:
no panic and data has been packed into bytes array
### what I actually get:
```
Running tool: /usr/local/go/bin/go test -timeout 30s -run ^TestWriteDeltaBitPackedInt64$ github.com/apache/arrow/go/v13/parquet/internal/encoding -v
=== RUN TestWriteDeltaBitPackedInt64
=== RUN TestWriteDeltaBitPackedInt64/panic_data
--- FAIL: TestWriteDeltaBitPackedInt64 (0.00s)
--- FAIL: TestWriteDeltaBitPackedInt64/panic_data (0.00s)
panic: runtime error: slice bounds out of range [:1026] with capacity 1024 [recovered]
panic: runtime error: slice bounds out of range [:1026] with capacity 1024
goroutine 19 [running]:
testing.tRunner.func1.2({0x103b6c000, 0x1400019c618})
/usr/local/go/src/testing/testing.go:1526 +0x1c8
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1529 +0x364
panic({0x103b6c000, 0x1400019c618})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/apache/arrow/go/v13/parquet/internal/encoding.(*deltaBitPackEncoder).FlushValues(0x140001eec00)
/Users/illyrix/Workspace/arrow/go/parquet/internal/encoding/delta_bit_packing.go:457 +0x33c
github.com/apache/arrow/go/v13/parquet/internal/encoding_test.TestWriteDeltaBitPackedInt64.func1(0x0?)
/Users/illyrix/Workspace/arrow/go/parquet/internal/encoding/encoding_test.go:642 +0xac
testing.tRunner(0x14000185ba0, 0x14000401680)
/usr/local/go/src/testing/testing.go:1576 +0x104
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1629 +0x370
FAIL github.com/apache/arrow/go/v13/parquet/internal/encoding 0.976s
```
--------------
there is another issue about delta_bit_packing( #35718 ), but it may be a different bug. All values in our test case are non-null.
### Component(s)
Go, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #37102: [Go] [Parquet] panic when writing particular dataset encoded by DeltaBitPacked
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #37102:
URL: https://github.com/apache/arrow/issues/37102#issuecomment-1673503586
cc @zeroshade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade closed issue #37102: [Go][Parquet] panic when writing particular dataset encoded by DeltaBitPacked
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #37102: [Go][Parquet] panic when writing particular dataset encoded by DeltaBitPacked
URL: https://github.com/apache/arrow/issues/37102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #37102: [Go] [Parquet] panic when writing particular dataset encoded by DeltaBitPacked
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #37102:
URL: https://github.com/apache/arrow/issues/37102#issuecomment-1673525931
Oh, i believe thats:
```go
// ReserveBytes reserves the next aligned nbytes, skipping them and returning
// the offset to use with WriteAt to write to those reserved bytes. Used for
// RLE encoding to fill in the indicators after encoding.
func (b *BitWriter) ReserveBytes(nbytes int) int {
b.Flush(true)
ret := b.byteoffset
b.byteoffset += nbytes
return ret
}
```
This doesn't extent `wr`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org