Re: [I] Support `VARIANT` type for unstructured data [datafusion]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on issue #16116:
URL: https://github.com/apache/datafusion/issues/16116#issuecomment-3340147780

   A brief update here -- as of Arrow 57.0.0, I think there will be enough Variant support in the parquet crate to actually use it in DataFusion and there is enough support, as described in [Implementing User Defined Types and Custom Metadata in DataFusion](https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata/) for extension types
   
   I would suggest starting off with the basic input/output:
   
   Add some udfs for calling the arrow kernels
   1. `variant_to_json` to convert a variant column to String (and JSON)
   2. `json_to_variant`: the opposite of the above
   3. `variant_get`: Extract subpaths
   4. `cast_to_variant`: casting columns to variant
   
   I don't think any of this code should be in the "core" datafusion crate, and we could make a `datafusion-variant` crate either in the main repo, or maybe even in `datafusion-contrib`
   
   I am hoping that any features driven by variant (e.g. user defined casting for example) would result in APIs in the core that are implemented by the variant crate
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org