You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Hattonuri (via GitHub)" <gi...@apache.org> on 2023/10/23 13:47:16 UTC
[I] [Go] DeltaBitPack deltaBitWidths overflow [arrow]
Hattonuri opened a new issue, #38399:
URL: https://github.com/apache/arrow/issues/38399
### Describe the bug, including details regarding any error messages, version, and platform.
I wrote some int64 using c++ ::parquet::Encoding::DELTA_BINARY_PACKED
and then tried to read this with go but got:
```
panic: runtime error: index out of range [4] with length 4
goroutine 199 [running]:
github.com/apache/arrow/go/v14/parquet/internal/encoding.(*DeltaBitPackInt64Decoder).unpackNextMini(0xc1aee302a0)
/home/dstasenko/go/pkg/mod/github.com/apache/arrow/go/v14@v14.0.0-20231023113027-37935604bf16/parquet/internal/encoding/delta_bit_packing.go:230 +0x2da
github.com/apache/arrow/go/v14/parquet/internal/encoding.(*DeltaBitPackInt64Decoder).Decode(0xc1aee302a0, {0xc1cc73b008?, 0xa7d50d?, 0xc1adb08360?})
/home/dstasenko/go/pkg/mod/github.com/apache/arrow/go/v14@v14.0.0-20231023113027-37935604bf16/parquet/internal/encoding/delta_bit_packing.go:267 +0x118
github.com/apache/arrow/go/v14/parquet/file.(*Int64ColumnChunkReader).ReadBatch.func1(0x15a201, 0x167d01)
/home/dstasenko/go/pkg/mod/github.com/apache/arrow/go/v14@v14.0.0-20231023113027-37935604bf16/parquet/file/column_reader_types.gen.go:93 +0xa3
github.com/apache/arrow/go/v14/parquet/file.(*columnChunkReader).readBatch(0xc1adb08360, 0x3c9c00, {0xc1d3f26000, 0x3c9c00, 0x3c9c00}, {0x0, 0x0, 0x0}, 0xc1aeb73c80)
/home/dstasenko/go/pkg/mod/github.com/apache/arrow/go/v14@v14.0.0-20231023113027-37935604bf16/parquet/file/column_reader.go:508 +0xc2
github.com/apache/arrow/go/v14/parquet/file.(*Int64ColumnChunkReader).ReadBatch(0xc1aee2e3d0?, 0xc1aee36960?, {0xc1cbc6a000?, 0x2b21d60?, 0xc0001a9d58?}, {0xc1d3f26000?, 0x1a49d55?, 0xa?}, {0x0, 0x0, ...})
```
I assume that this line lacks boundaries check https://github.com/apache/arrow/blob/ac581fd2a87b35c872cf334bb147851fe1287714/go/parquet/internal/encoding/delta_bit_packing.go#L228
### Component(s)
Go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] DeltaBitPack deltaBitWidths overflow [arrow]
Posted by "Hattonuri (via GitHub)" <gi...@apache.org>.
Hattonuri commented on issue #38399:
URL: https://github.com/apache/arrow/issues/38399#issuecomment-1775436309
[storage_000000000.parquet.txt](https://github.com/apache/arrow/files/13072508/storage_000000000.parquet.txt)
added .txt ending because github does not allow to upload parquet files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] DeltaBitPack deltaBitWidths overflow [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #38399:
URL: https://github.com/apache/arrow/issues/38399#issuecomment-1775563734
I've found `if err != nil return`, but it's still propogate an error
C++ is able to read file like this, let me investigate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] DeltaBitPack deltaBitWidths overflow [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #38399: [Go] DeltaBitPack deltaBitWidths overflow
URL: https://github.com/apache/arrow/issues/38399
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] DeltaBitPack deltaBitWidths overflow [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #38399:
URL: https://github.com/apache/arrow/issues/38399#issuecomment-1775600967
I've find out the reason. Let me submit a fixing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] DeltaBitPack deltaBitWidths overflow [arrow]
Posted by "Hattonuri (via GitHub)" <gi...@apache.org>.
Hattonuri commented on issue #38399:
URL: https://github.com/apache/arrow/issues/38399#issuecomment-1775446441
https://github.com/apache/arrow/blob/ac581fd2a87b35c872cf334bb147851fe1287714/go/parquet/internal/encoding/delta_bit_packing.go#L175-L176
probably you forgot "if err != nil return"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] DeltaBitPack deltaBitWidths overflow [arrow]
Posted by "Hattonuri (via GitHub)" <gi...@apache.org>.
Hattonuri commented on issue #38399:
URL: https://github.com/apache/arrow/issues/38399#issuecomment-1777150461
Now it works! Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] DeltaBitPack deltaBitWidths overflow [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #38399:
URL: https://github.com/apache/arrow/issues/38399#issuecomment-1775607757
@Hattonuri I've submit a fixing here: https://github.com/apache/arrow/pull/38413
You can have a try at the new code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] DeltaBitPack deltaBitWidths overflow [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #38399:
URL: https://github.com/apache/arrow/issues/38399#issuecomment-1775331602
Would you mind provide the test data which might trigger the bug here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Go] DeltaBitPack deltaBitWidths overflow [arrow]
Posted by "Hattonuri (via GitHub)" <gi...@apache.org>.
Hattonuri commented on issue #38399:
URL: https://github.com/apache/arrow/issues/38399#issuecomment-1777118703
Now it works! Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org