You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "wjones127 (via GitHub)" <gi...@apache.org> on 2024/06/18 16:52:32 UTC
[I] Open Variant Type for semi-structured data [datafusion]
wjones127 opened a new issue, #10987:
URL: https://github.com/apache/datafusion/issues/10987
I've been starting to experiment with implementing the Open Variant Type [^1] in Rust / DataFusion. There is a specification and Java library for this, and Spark will release this type in 4.0. There are also plans to integrate this into table formats such as Delta Lake [^3] and Iceberg [^4]. This would be a high-performance data type for semi-structured data, designed for better OLAP performance than JSON or BSON (discussed in #7845). I've discussed a little bit in the Arrow repo about it's potential as an Arrow extension type [^2].
I'm working on creating an extension similar to [datafusion-functions-json](https://github.com/datafusion-contrib/datafusion-functions-json). If we could create a new repo `datafusion-functions-variant`, I'd be happy to develop that in the open.
[^1]: https://github.com/apache/spark/tree/master/common/variant
[^2]: https://github.com/apache/arrow/issues/42069
[^3]: https://www.databricks.com/blog/introducing-open-variant-data-type-delta-lake-and-apache-spark
[^4]: https://lists.apache.org/thread/xnyo1k66dxh0ffpg7j9f04xgos0kwc34
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Open Variant Type for semi-structured data [datafusion]
Posted by "samuelcolvin (via GitHub)" <gi...@apache.org>.
samuelcolvin commented on issue #10987:
URL: https://github.com/apache/datafusion/issues/10987#issuecomment-2176987939
Interesting, this is exactly what we need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Open Variant Type for semi-structured data [datafusion]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on issue #10987:
URL: https://github.com/apache/datafusion/issues/10987#issuecomment-2892638977
There's a discussion happening on the Arrow dev mailing list about adding Variant as a canonical extension type in the Arrow spec. Input from the DataFusion developer community would be much appreciated.
- The discussion thread in the mailing list archive: https://lists.apache.org/thread/w06cxdojjcmry4m9vb0bo7owd1jsbtz5
- How to subscribe to the Arrow dev mailing list: https://arrow.apache.org/community/#mailing-lists
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Open Variant Type for semi-structured data [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10987:
URL: https://github.com/apache/datafusion/issues/10987#issuecomment-2176979863
@samuelcolvin and @adriangb and I were speaking about datafusion-functions-json just today. They may be interested too
I have created https://github.com/datafusion-contrib/datafusion-functions-variant and added you as an admin 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Open Variant Type for semi-structured data [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10987:
URL: https://github.com/apache/datafusion/issues/10987#issuecomment-2836521200
FWIW the variant type has been added to the parquet spec, and is now much closer to standardization and we are beginning work on the support at the parquet layer. See epic
- https://github.com/apache/arrow-rs/issues/6736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org