You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "thorfour (via GitHub)" <gi...@apache.org> on 2023/05/16 16:49:25 UTC
[GitHub] [arrow] thorfour opened a new issue, #35621: [Go] IPC Writer encodes sliced list arrays incorrectly.
thorfour opened a new issue, #35621:
URL: https://github.com/apache/arrow/issues/35621
### Describe the bug, including details regarding any error messages, version, and platform.
When you create a record with a list column, if you slice that array and then try and encode the slices with the IPC writer, it produces incorrect results when read back from the IPC format.
It appears to incorrectly address the offsets of a slice if the slice has a non-zero offset. Below is an example test that fails because the second slice returns the first two lists after IPC encoding instead of the expected last two lists.
```
func Test_Arrow_SlicedList(t *testing.T) {
schema := arrow.NewSchema([]arrow.Field{
{
Name: "letters",
Type: arrow.ListOf(&arrow.StringType{}),
},
}, nil)
rb := array.NewRecordBuilder(memory.NewGoAllocator(), schema)
lb := rb.Field(0).(*array.ListBuilder)
sb := lb.ValueBuilder().(*array.StringBuilder)
lb.Append(true)
sb.Append("a")
sb.Append("b")
sb.Append("c")
lb.Append(true)
sb.Append("d")
sb.Append("e")
sb.Append("f")
lb.Append(true)
sb.Append("g")
sb.Append("h")
sb.Append("i")
r := rb.NewRecord()
r0 := r.NewSlice(0, 1)
r1 := r.NewSlice(1, 3)
b := &bytes.Buffer{}
w := ipc.NewWriter(b, ipc.WithSchema(r.Schema()))
require.NoError(t, w.Write(r0))
require.NoError(t, w.Write(r1))
require.NoError(t, w.Close())
rdr, err := ipc.NewReader(b)
require.NoError(t, err)
rdr.Next()
read0 := rdr.Record()
read0.Retain()
rdr.Next()
read1 := rdr.Record()
require.True(t, array.RecordEqual(r0, read0))
require.True(t, array.RecordEqual(r1, read1))
}
```
```
thor@thors-MacBook-Pro frostdb % go test -v -run SlicedList
=== RUN Test_Arrow_SlicedList
table_test.go:1773:
Error Trace: /Users/thor/go/src/github.com/polarsignals/frostdb/table_test.go:1773
Error: Should be true
Test: Test_Arrow_SlicedList
--- FAIL: Test_Arrow_SlicedList (0.00s)
```
### Component(s)
Go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thorfour closed issue #35621: [Go] IPC Writer encodes sliced list arrays incorrectly.
Posted by "thorfour (via GitHub)" <gi...@apache.org>.
thorfour closed issue #35621: [Go] IPC Writer encodes sliced list arrays incorrectly.
URL: https://github.com/apache/arrow/issues/35621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on issue #35621: [Go] IPC Writer encodes sliced list arrays incorrectly.
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #35621:
URL: https://github.com/apache/arrow/issues/35621#issuecomment-1550042685
@thorfour which version of Arrow are you using here? I ask because issues with IPC slicing list/map arrays were brought up in #14780 and addressed by #14793. That fix went out with Arrow v11 so if you're using an earlier version than Go Arrow v11, can you try upgrading first and seeing if you can still reproduce the failure in v11 or v12?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thorfour commented on issue #35621: [Go] IPC Writer encodes sliced list arrays incorrectly.
Posted by "thorfour (via GitHub)" <gi...@apache.org>.
thorfour commented on issue #35621:
URL: https://github.com/apache/arrow/issues/35621#issuecomment-1550109873
Wow thank you for pointing that out! I thought I had updated my version, but apparently my go.mod was incorrect. It's working as expected with the latest. Thank you so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thorfour commented on issue #35621: [Go] IPC Writer encodes sliced list arrays incorrectly.
Posted by "thorfour (via GitHub)" <gi...@apache.org>.
thorfour commented on issue #35621:
URL: https://github.com/apache/arrow/issues/35621#issuecomment-1550342046
I'm reopening this because I discovered that the bug still exists for `LargeLists` but verified that it indeed worked as expected for regular lists
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thorfour commented on issue #35621: [Go] IPC Writer encodes sliced list arrays incorrectly.
Posted by "thorfour (via GitHub)" <gi...@apache.org>.
thorfour commented on issue #35621:
URL: https://github.com/apache/arrow/issues/35621#issuecomment-1551452918
Okay, closing again because it looks like that was fixed in v13 now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] thorfour closed issue #35621: [Go] IPC Writer encodes sliced list arrays incorrectly.
Posted by "thorfour (via GitHub)" <gi...@apache.org>.
thorfour closed issue #35621: [Go] IPC Writer encodes sliced list arrays incorrectly.
URL: https://github.com/apache/arrow/issues/35621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org