[GitHub] [pinot] richardstartin edited a comment on issue #7881: Support FLATTEN (unnest), KVGEN type functions

[GitBox <gi...@apache.org>]

richardstartin edited a comment on issue #7881:
URL: https://github.com/apache/pinot/issues/7881#issuecomment-988730649


   This may require a table or dataframe abstraction at the block level and the ability to join the table to other blocks on docId. Imagine you have documents at docId N and N+1
   
   ```
   {
      "name": "abc",
      "age": 25,
      "phones": [123, 456, 789]
   }, <- row N
   {
      "name": "xyz",
      "age": 25,
      "phones": [321, 654, 987]
   } <- row N+1
   ```
   
   which is flattened to 
   
   ```
   abc 25 123 N
   abc 25 456 N
   abc 25 789 N
   xyz 25 321 N+1
   xyz 25 654 N+1
   xyz 25 987 N+1
   ```
   
   and if you want to group by age and some other field not in the JSON, then it needs to be joined to a block
   
   ```
   foo <- row N
   bar <- row N+1
   ```
   
   to create a wide table
   
   ```
   abc 25 123 foo
   abc 25 456 foo
   abc 25 789 foo
   xyz 25 321 bar
   xyz 25 654 bar
   xyz 25 987 bar
   ```
   
   which can then be provided as the input to group by queries and so on.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org