[GitHub] [incubator-pinot] kishoreg opened a new pull request #4388: Adding support for Map type fields

[GitBox <gi...@apache.org>]

kishoreg opened a new pull request #4388: Adding support for Map type fields
URL: https://github.com/apache/incubator-pinot/pull/4388
 
 
   This leverages existing features MultiValue feature in Pinot. A Map is represented as two Multi-valued columns - __KEYS and __VALUES. This convention is used in only one place - real-time ingestion and segment generation. When pinot sees a FieldSpec name that ends with __KEYS or __VALUES, it checks for the field in the input Record. If that object turns out to be a map, then we convert it into two columns -> keyArray and valueArray.
   
   Once we ingest a map as two arrays, we can access map['keyName']  via udf map_value(keyColumnName, 'KeyName', valueColumnName). We can support the simpler syntax when we move to calcite.
   
   Sample queries
   
   ```select map_value(myMap__KEYS, 'k1', myMap__VALUES) from myTable```
   
   ```select count(*) from myTable group by map_value(myMap__KEYS, 'k1', myMap__VALUES)```
   
   ```select count(*) from myTable where map_value(myMap__KEYS, 'k1', myMap__VALUES) = 'v1'```
   
   NOTE:
   While this solution is not elegant, it is a good first step towards supporting Map and Struct data types.
   Supporting Map as a DataType was bit invasive and requires us to first introduce the concept of CompositeColumnDatasource.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org