[GitHub] [pinot] kirkrodrigues commented on issue #9819: Add plugins to support storing and querying JSON log events in Pinot

[GitBox <gi...@apache.org>]

kirkrodrigues commented on issue #9819:
URL: https://github.com/apache/pinot/issues/9819#issuecomment-1319521363

   The plugins would be limited to input JSON records and CLP's encoding can be applied to any text field within those records.
   
   Our short term goal is to kind of use Pinot as a black box columnar store. So we'd apply CLP's encoding to decompose a text field into a columnar format and store the columns in Pinot; for logs this should reduce the storage overhead of that field while still allowing it to be searched without resorting to a text index. Then when a user wants to query the field, we'd use CLP to convert their wildcard query into a SQL query on the decomposed columns in Pinot. Since the query operates on the decomposed columns, this should be faster than a query on the original text and only matching rows would need to be reconstructed from the columns using a UDF.
   
   If this works well, (and if the community gives us their blessing :), we hope to try and integrate this deeper into Pinot, perhaps as a special type of index that could be applied to any text column which contains logs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org