[GitHub] [arrow-julia] omus commented on issue #367: inappropriately applied `Arrow.NullVector` optimization to `Union{ZonedDateTime,Missing}` column

[GitBox <gi...@apache.org>]

omus commented on issue #367:
URL: https://github.com/apache/arrow-julia/issues/367#issuecomment-1371389919

   > - In general, I wonder if it'd be better for a Timestamp's time zone to be a value rather than "lifted" into the type domain. idk what restrictions are being imposed on us by the Arrow format itself, though.
   
   Our timestamp implementation of `Timestamp{U, TZ}` where `U` is the unit precision and `TZ` is the name of the time zone is very closely aligned to the pyarrow implementation: https://arrow.apache.org/docs/python/generated/pyarrow.timestamp.html#pyarrow.timestamp. Looking at the [Arrow specification for a `Timestamp`](https://github.com/apache/arrow/blob/c6eb4aad30280e1df269f526de057a479c5bc68a/format/Schema.fbs#L233-L352) also confirms this is the underlying representation. One repercussion of this to note is you cannot convert a vector of `ZonedDateTime`s with mixed time zones into the Arrow Timestamp representation.
   
   Due to `ZonedDateTime` contains the time zone name within instances of the type we cannot define our `ArrowType(::Type{ZonedDateTime})` to return a concrete type as long as we keep `TZ` as a type parameter in `Timestamp`. We could remove `TZ` as a type parameter to resolve this but we'd end up running into issues with our `Timestamp` Julia type appearing that multiple time zones could be supported in single column.
   
   > One thing we could do to improve the handrolled inference might be to always generate an Arrow.default for every concrete Julia type present in the input eltype, and always include those values' types in our starting T before the promoteunion loop?
   
   Generating the `ToArrow` eltype with `Arrow.default` as the starting `T` problematic as we cannot support multiple time zones within the same vector. Always using a default would result in `"UTC"` being the only allowed time zone. What we can do however is fall back on using `default` when the inferred eltype is `Missing`. The result of is that `default` is only called when all elements of the vector are `missing` and the union type contains a non-concrete type.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org