You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "ericphanson (via GitHub)" <gi...@apache.org> on 2023/06/06 15:59:44 UTC
[GitHub] [arrow-julia] ericphanson opened a new issue, #452: Arrow.jl 2.6 breaks Legolas.jl's tests
ericphanson opened a new issue, #452:
URL: https://github.com/apache/arrow-julia/issues/452
As well as Onda.jl
E.g. [this test](https://github.com/beacon-biosignals/Legolas.jl/blob/321b19c88c5c7e07c616278f2aaf876495533667/test/runtests.jl#L459-L468) fails with the error:
```julia
┌ Warning: The `Tables.Schema` of the `Arrow.Table` read via `Legolas.read(io_or_path)` does not appear to
│ comply with the `Legolas.SchemaVersion` indicated by the table's metadata (`SchemaVersion("test.parent", 1)`). Try invoking
│ `Legolas.read(io_or_path; validate=false)` to inspect the table.
└ @ Legolas ~/Legolas.jl/src/tables.jl:171
miscellaneous Legolas/src/tables.jl tests: Error During Test at /Users/eph/Legolas.jl/test/runtests.jl:468
Test threw exception
Expression: t == [NamedTuple(ParentV1(r)) for r = Tables.rows(Legolas.read(path))]
ArgumentError: Tables.Schema violates Legolas schema `test.parent@1`:
- Incorrect type: `x` expected `<:Vector`, found `SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}`
Provided Tables.Schema:
:x SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}
:y String
Stacktrace:
[1] validate(ts::Tables.Schema{(:x, :y), Tuple{SubArray{Int64, 1, Arrow.Primitive{Int64, Vector{Int64}}, Tuple{UnitRange{Int64}}, true}, String}}, sv::SchemaVersion{Symbol("test.parent"), 1})
@ Legolas ~/Legolas.jl/src/schemas.jl:312
[2] read(io_or_path::MyPath; validate::Bool)
@ Legolas ~/Legolas.jl/src/tables.jl:169
[3] read(io_or_path::MyPath)
@ Legolas ~/Legolas.jl/src/tables.jl:159
[4] macro expansion
@ ~/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.8/Test/src/Test.jl:464 [inlined]
[5] macro expansion
@ ~/Legolas.jl/test/runtests.jl:468 [inlined]
[6] macro expansion
@ ~/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.8/Test/src/Test.jl:1363 [inlined]
[7] top-level scope
@ ~/Legolas.jl/test/runtests.jl:459
┌ Warning: The `Tables.Schema` of the `Arrow.Table` read via `Legolas.read(io_or_path)` does not appear to
│ comply with the `Legolas.SchemaVersion` indicated by the table's metadata (`SchemaVersion("haha", 3)`). Try invoking
│ `Legolas.read(io_or_path; validate=false)` to inspect the table.
└ @ Legolas ~/Legolas.jl/src/tables.jl:171
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] ericphanson commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579044664
Maybe we should yank v2.6 from General? Since it wasn't meant to be a breaking release. cc @quinnj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] ericphanson closed issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson closed issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
URL: https://github.com/apache/arrow-julia/issues/452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] ericphanson commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579235646
Ok let's close this one then, thanks for the help everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] quinnj commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "quinnj (via GitHub)" <gi...@apache.org>.
quinnj commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579172525
FYI, we tweaked the definition [here](https://github.com/apache/arrow-julia/blob/8aed96998ace1010580a74d03222770140acd579/src/append.jl#L197) to accommodate the chance that list-type columns will return `SubArray` now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] hannahilea commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "hannahilea (via GitHub)" <gi...@apache.org>.
hannahilea commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579227113
Thanks for the clarification, all. I opened https://github.com/beacon-biosignals/Legolas.jl/pull/89 with the fix proposed by @ericphanson, since it (now!) seems clearer that the failing Legolas tests are due to Legolas being too(?) strict to begin with, rather than any change introduced by Arrow 2.6.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] Moelf commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "Moelf (via GitHub)" <gi...@apache.org>.
Moelf commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579095707
What's this actually testing, the :< Vector seems way too strong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] ericphanson commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579109900
I think it calls `convert` first, but then, yeah, it expects a `Vector{Int}` for the element (not for the whole column, just for the individual element). That is what we got on Arrow 2.5:
```julia
julia> Arrow.Table(read(path))
Arrow.Table with 2 rows, 2 columns, and schema:
:x Vector{Int64} (alias for Array{Int64, 1})
:y String
with metadata given by a Base.ImmutableDict{String, String} with 1 entry:
"legolas_schema_qualified" => "test.parent@1"
julia> Arrow.Table(read(path)).x
2-element Arrow.List{Vector{Int64}, Int32, Arrow.Primitive{Int64, Vector{Int64}}}:
[1, 2]
[3, 4]
julia> Arrow.Table(read(path)).x |> typeof
Arrow.List{Vector{Int64}, Int32, Arrow.Primitive{Int64, Vector{Int64}}}
julia> Arrow.Table(read(path)).x[1] |> typeof
Vector{Int64} (alias for Array{Int64, 1})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] Moelf commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "Moelf (via GitHub)" <gi...@apache.org>.
Moelf commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579115921
A column that's vector of vector (that doesn't have to be square)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] ericphanson commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579114608
What is a jagged column?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] ericphanson commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "ericphanson (via GitHub)" <gi...@apache.org>.
ericphanson commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579137402
Hm ok. I think we should be trying to convert at least before validating, I agree it is very strict otherwise. Legolas has a mechanism for related things, I think maybe we need to add some definitions like this:
```julia
Legolas.accepted_field_type(::Legolas.SchemaVersion, ::Type{Vector{String}}) = AbstractVector{<:AbstractString}
Legolas.accepted_field_type(::Legolas.SchemaVersion, ::Type{Vector{T}}) where T = AbstractVector{<:T}
Legolas.accepted_field_type(::Legolas.SchemaVersion, ::Type{Vector}) = AbstractVector
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] Moelf commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "Moelf (via GitHub)" <gi...@apache.org>.
Moelf commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579113922
Yeah well that's too strong, when you have a jagged column, it's best to not return Vector for each element because that copies.
Arrow stores jagged column as one data column and one offset column, like VectorOfVectors from (ArratOrArrays.jl). So really the promise is that each element will be :<AbstractVector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-julia] Moelf commented on issue #452: Arrow.jl 2.6 breaks Legolas.jl's tests
Posted by "Moelf (via GitHub)" <gi...@apache.org>.
Moelf commented on issue #452:
URL: https://github.com/apache/arrow-julia/issues/452#issuecomment-1579159646
Yeah, in any case Arrow schema is just a logical construction, most likely can't be mapped to Julia types one to one and shouldn't (other than the primitive bit types)
So we need some kind of Julia side logical schema in the end. (For vector it's nicely covered by AbstractVector, because Julia's type hierarchy is kinda designed for these technical usage)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org