You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/05/12 22:20:17 UTC
[GitHub] [incubator-pinot] yupeng9 opened a new issue #6904: Complex type handling
yupeng9 opened a new issue #6904:
URL: https://github.com/apache/incubator-pinot/issues/6904
Today Pinot supports primitive data types (including int, long, float, double, string, bytes), as well as limited multi-value types such as an array of primitive types. Such simple data types allow Pinot to build fast indexing structures for good query performance, but with a tradeoff on the ingestion limitations. It’s common for users to ingest data with complex data types into Pinot, such as JSON or Avro, in particular with two complex types: tuple (.e.g record in Avro, object in JSON) and list of complex data types (e.g. arrays in Avro and JSON). However, today most users have to set up transformation pipelines, e.g. streaming pipeline in Flink for real-time ingestion or batch pipeline in Spark for offline ingestion, in order to flatten/transform the complex data types in the primitives ones to be compatible with Pinot’s data types. Many users find authoring such pipelines is difficult, especially when the schema is complex, for example, deeply nested.
A proposal to address this issue is at this [doc](https://docs.google.com/document/d/1L6d0k_yF4TbFWHt5IJ5eYYjJZboRnJTBrQYRB2an9iI/edit#)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 closed issue #6904: Complex type handling
Posted by GitBox <gi...@apache.org>.
yupeng9 closed issue #6904:
URL: https://github.com/apache/incubator-pinot/issues/6904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on issue #6904: Complex type handling
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on issue #6904:
URL: https://github.com/apache/incubator-pinot/issues/6904#issuecomment-849938401
User docs is at https://docs.pinot.apache.org/basics/data-import/complex-type
This will be released as part of 0.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org