[GitHub] [incubator-pinot] kkrugler commented on issue #6473: Support no forward index for column

[GitBox <gi...@apache.org>]

kkrugler commented on issue #6473:
URL: https://github.com/apache/incubator-pinot/issues/6473#issuecomment-782425259


   Hi @siddharthteotia - yes, one example segment is 2,637,935 rows, and `metadata.properties` for the column of interest (`creativeText_terms`) has cardinality of 48,591 (though that's lower than what I was expecting).
   
   ```
   column.creativeText_terms.cardinality = 48591
   column.creativeText_terms.totalDocs = 2637935
   column.creativeText_terms.dataType = STRING
   column.creativeText_terms.bitsPerElement = 16
   column.creativeText_terms.lengthOfEachEntry = 60
   column.creativeText_terms.columnType = DIMENSION
   column.creativeText_terms.isSorted = false
   column.creativeText_terms.hasNullValue = false
   column.creativeText_terms.hasDictionary = true
   column.creativeText_terms.textIndexType = NONE
   column.creativeText_terms.hasInvertedIndex = true
   column.creativeText_terms.hasFSTIndex = false
   column.creativeText_terms.hasJsonIndex = false
   column.creativeText_terms.isSingleValues = false
   column.creativeText_terms.maxNumberOfMultiValues = 49
   column.creativeText_terms.totalNumberOfEntries = 14628086
   column.creativeText_terms.isAutoGenerated = false
   column.creativeText_terms.minValue = 0.01
   column.creativeText_terms.maxValue = \u1EE9ng
   column.creativeText_terms.defaultNullValue = null
   ```
   
   The dictionary is 2.9MB, and the forward index is 31MB:
   
   ```
   creativeText_terms.dictionary.startOffset = 1648876
   creativeText_terms.dictionary.size = 2915468
   creativeText_terms.forward_index.startOffset = 4564344
   creativeText_terms.forward_index.size = 31110427
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org