Re: [I] colmetadata does not read custom metadata with multiple writes [arrow-julia]

["ericphanson (via GitHub)" <gi...@apache.org>]

ericphanson commented on issue #485:
URL: https://github.com/apache/arrow-julia/issues/485#issuecomment-4099596822

   ```julia
   
   using Arrow
   using Tables
   
   # Write an Arrow file with table metadata and column metadata.
   # The same schema is written in multiple record batches when nbatches > 1.
   function write_issue485_file(path; nbatches::Int)
       rm(path; force=true)
   
       filemeta = ["file" => path, "note" => "issue #485 repro"]
       colmeta1 = ["id" => "chan1", "comment" => "Column1 metadata"]
       colmeta2 = ["id" => "chan2", "comment" => "Column2 metadata"]
   
       writer = open(
           Arrow.Writer,
           path;
           metadata=filemeta,
           colmetadata=Dict(:Column1 => colmeta1, :Column2 => colmeta2),
       )
   
       try
           for i in 1:nbatches
               # Use the same write path as the original issue:
               # a matrix converted with Tables.table(...)
               data = rand(2, 2) .+ i
               Arrow.write(writer, Tables.table(data))
           end
       finally
           close(writer)
       end
   end
   
   # Read the file back and print metadata at both levels:
   #   1) the top-level combined Arrow.Table column
   #   2) each partition / record batch separately
   function inspect_file(path)
       tbl = Arrow.Table(path)
   
       println("file: ", path)
       println("typeof(tbl.Column1): ", typeof(tbl.Column1))
       println("table metadata: ", Arrow.getmetadata(tbl))
       println("top-level Column1 metadata: ", Arrow.getmetadata(tbl.Column1))
       println("top-level Column2 metadata: ", Arrow.getmetadata(tbl.Column2))
   
       # Partition-local columns keep their metadata even in the failing case.
       println("partition-local Column1 metadata:")
       for (i, part) in enumerate(Tables.partitions(tbl))
           println("  partition ", i, ": ", Arrow.getmetadata(part.Column1))
       end
       println()
   end
   ```
   
   gives
   
   ```julia
   julia> write_issue485_file("issue485_one.arrow"; nbatches=1)
   
   julia> inspect_file("issue485_one.arrow")
   file: issue485_one.arrow
   typeof(tbl.Column1): Arrow.Primitive{Float64, Vector{Float64}}
   table metadata: Base.ImmutableDict("note" => "issue #485 repro", "file" => "issue485_one.arrow")
   top-level Column1 metadata: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
   top-level Column2 metadata: Base.ImmutableDict("comment" => "Column2 metadata", "id" => "chan2")
   partition-local Column1 metadata:
     partition 1: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
   ```
   
   so things work as expected with top-level column metadata when we have 1 batch, but with 2:
   
   ```julia
   julia> write_issue485_file("issue485_two.arrow"; nbatches=2)
   
   julia> inspect_file("issue485_two.arrow")
   file: issue485_two.arrow
   typeof(tbl.Column1): SentinelArrays.ChainedVector{Float64, Arrow.Primitive{Float64, Vector{Float64}}}
   table metadata: Base.ImmutableDict("note" => "issue #485 repro", "file" => "issue485_two.arrow")
   top-level Column1 metadata: nothing
   top-level Column2 metadata: nothing
   partition-local Column1 metadata:
     partition 1: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
     partition 2: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
   ```
   
   now the top-level column metadata is nothing, but the individual partitions still have the metadata.
   
   I think there is a missing `getmetadata` method, e.g. this works:
   
   
   ```julia
   julia> using Arrow, SentinelArrays
   
   julia> function Arrow.getmetadata(x::SentinelArrays.ChainedVector{T,A}) where {T,A<:Arrow.ArrowVector}
              isempty(x.arrays) && return nothing
   
              m0 = Arrow.getmetadata(first(x.arrays))
              for a in Iterators.drop(x.arrays, 1)
                  m = Arrow.getmetadata(a)
                  if m != m0
                      throw(ArgumentError("inconsistent column metadata across ChainedVector chunks"))
                  end
              end
              return m0
          end
   getmetadata (generic function with 5 methods)
   
   julia> inspect_file("issue485_two.arrow")
   file: issue485_two.arrow
   typeof(tbl.Column1): ChainedVector{Float64, Arrow.Primitive{Float64, Vector{Float64}}}
   table metadata: Base.ImmutableDict("note" => "issue #485 repro", "file" => "issue485_two.arrow")
   top-level Column1 metadata: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
   top-level Column2 metadata: Base.ImmutableDict("comment" => "Column2 metadata", "id" => "chan2")
   partition-local Column1 metadata:
     partition 1: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
     partition 2: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org