You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Moelf (via GitHub)" <gi...@apache.org> on 2023/04/07 03:30:29 UTC
[GitHub] [arrow-julia] Moelf opened a new issue, #418: `eachindex()` on columns producing unexpected result
Moelf opened a new issue, #418:
URL: https://github.com/apache/arrow-julia/issues/418
`eachindex()` can return useful index or garbage depending on the size of the Arrow file:
```julia
julia> Arrow.append("./out2.feather", gendata())
"./out2.feather"
julia> tbl2 = Arrow.Table("./out2.feather")
Arrow.Table with 3 rows, 2 columns, and schema:
:x Vector{Float64} (alias for Array{Float64, 1})
:y Vector{Float64} (alias for Array{Float64, 1})
julia> tbl2.x
3-element Arrow.List{Vector{Float64}, Int32, Arrow.Primitive{Float64, Vector{Float64}}}:
[0.38906822055013135]
[0.9737332599073911, 0.19757843080977833, 0.06268161159155239, 0.9716472746828138, 0.8838743377016612, 0.951662206335832, 0.5973352614611379, 0.8876145238421496, 0.5149018084376046]
[0.9363691598185577, 0.3603564376862347, 0.6425331531623392, 0.4465148469221817, 0.9705740924873874, 0.7543076133441144, 0.9591142326905828, 0.1521499948934163, 0.49812895258696377]
julia> eachindex(tbl2.x)
Base.OneTo(3)
```
```julia
julia> foreach(1:100) do _
Arrow.append("./out2.feather", gendata())
end
julia> tbl2 = Arrow.Table("./out2.feather")
Arrow.Table with 303 rows, 2 columns, and schema:
:x Vector{Float64} (alias for Array{Float64, 1})
:y Vector{Float64} (alias for Array{Float64, 1})
julia> eachindex(tbl2.x)
SentinelArrays.IndexIterator{Arrow.List{Vector{Float64}, Int32, Arrow.Primitive{Float64, Vector{Float64}}}}(Arrow.List{Vector{Float64}, Int32, Arrow.Primitive{Float64, Vector{Float64}}}[[[0.38906822055013135], [0.9737332599073911, 0.19757843080977833, 0.06268161159155239, 0.9716472746828138, 0.8838743377016612, 0.951662206335832, 0.5973352614611379, 0.8876145238421496, 0.5149018084376046], [0.9363691598185577, 0.3603564376862347, 0.6425331531623392, 0.4465148469221817, 0.9705740924873874, 0.7543076133441144, 0.9591142326905828, 0.1521499948934163, 0.49812895258696377]], [[0.723214841767352, 0.43489448845317247, 0.7435036925620425, 0.4503683881092432, 0.7568472306565802, 0.9434247930999995, 0.1137908376766733, 0.9561139152795906], [0.48214812381208816, 0.08258591275570137, 0.6524745829052587, 0.1384493817488982
```
I understand this is because SentinelArrays, but this is really unintuitive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] Moelf commented on issue #418: `eachindex()` on columns producing unexpected result
Posted by "Moelf (via GitHub)" <gi...@apache.org>.
Moelf commented on issue #418:
URL: https://github.com/apache/arrow-julia/issues/418#issuecomment-1577904476
because all file format doesn't have a "vanilla" way to express "these two jagged columns have same number of elements row by row", physicists in my field are somewhat used to doing this.\
maybe we can add a `parent() === ...` check in `getindex()`? it should get optimized away
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] Moelf commented on issue #418: `eachindex()` on columns producing unexpected result
Posted by "Moelf (via GitHub)" <gi...@apache.org>.
Moelf commented on issue #418:
URL: https://github.com/apache/arrow-julia/issues/418#issuecomment-1501219367
I don't mind what SentinelArrays.jl do and I get it's for performance.
But just the fact that given the same Arrow schema, the type of `eachindex()` of the same column changes depending on if the file contains 1 batch or more than 1 batch seems bad enough already
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] ericphanson commented on issue #418: `eachindex()` on columns producing unexpected result
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson commented on issue #418:
URL: https://github.com/apache/arrow-julia/issues/418#issuecomment-1501218646
Ah, yeah, that seems non-ideal, although maybe not directly a bug.
Maybe some options here are to change the return here, change that special index so it errors when used with another vector, or if neither of those can be done performantly, just try to document this behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] ericphanson commented on issue #418: `eachindex()` on columns producing unexpected result
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson commented on issue #418:
URL: https://github.com/apache/arrow-julia/issues/418#issuecomment-1501112789
What do you mean by garbage exactly, the verbose type? Does it work to actually index with the result? (Not sure if this is actually a bug)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] quinnj commented on issue #418: `eachindex()` on columns producing unexpected result
Posted by "quinnj (via GitHub)" <gi...@apache.org>.
quinnj commented on issue #418:
URL: https://github.com/apache/arrow-julia/issues/418#issuecomment-1577895111
Hmmmm, it seems pretty bad form though to be trying to use the indices of one array to index another? Like, I get that having a better error here would be nice, but that just seems like a really big no-no from the start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] Moelf commented on issue #418: `eachindex()` on columns producing unexpected result
Posted by "Moelf (via GitHub)" <gi...@apache.org>.
Moelf commented on issue #418:
URL: https://github.com/apache/arrow-julia/issues/418#issuecomment-1501204277
it works but users have to be extremely careful:
```julia
julia> [tbl2.x[i] for i in eachindex(tbl2.x)] == collect(tbl2.x)
true
julia> [tbl2.y[i] for i in eachindex(tbl2.y)] == collect(tbl2.y)
true
julia> [tbl2.y[i] for i in eachindex(tbl2.x)] == collect(tbl2.y)
false
julia> [tbl2.x[i] for i in eachindex(tbl2.y)] == collect(tbl2.x)
false
```
the last 2 lines break because `SentinelArrays.IndexIterator` which only happens when the table have multiple batches.
Notice, if the user remembered to use `eachindex(tbl2.x, tbl2.y)` it would normalize to `OneTo(6)` and everything would be fine. But the problem is neither of the last 2 lines threw error, just silently corrupted result
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] quinnj commented on issue #418: `eachindex()` on columns producing unexpected result
Posted by "quinnj (via GitHub)" <gi...@apache.org>.
quinnj commented on issue #418:
URL: https://github.com/apache/arrow-julia/issues/418#issuecomment-1579564488
Alright, I think this is a pretty easy assertion: https://github.com/JuliaData/SentinelArrays.jl/pull/90
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] quinnj closed issue #418: `eachindex()` on columns producing unexpected result
Posted by "quinnj (via GitHub)" <gi...@apache.org>.
quinnj closed issue #418: `eachindex()` on columns producing unexpected result
URL: https://github.com/apache/arrow-julia/issues/418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org