You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ksnijjer (via GitHub)" <gi...@apache.org> on 2023/12/28 05:14:01 UTC
[I] MV encoding config not working [pinot]
ksnijjer opened a new issue, #12207:
URL: https://github.com/apache/pinot/issues/12207
This is related to https://github.com/apache/pinot/pull/11993, while this config works
```
"fieldConfigList": [
{
"name": "col_name",
"encodingType": "DICTIONARY",
"compressionCodec": "MV_ENTRY_DICT"
}]
```
this format doesn't seem to work
```
"fieldConfigList": [
{
"name": "service_code",
"encodingType": "DICTIONARY",
"indexes": {
"forward": {
"dictIdCompressionType": "MV_ENTRY_DICT"
}
}
} ]
```
@Jackie-Jiang can we check on this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] MV encoding config not working [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #12207:
URL: https://github.com/apache/pinot/issues/12207#issuecomment-1961271770
I was able to reproduce the issue with the older Pinot version @ksnijjer was using, but not with the latest one. I think we can close it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] MV encoding config not working [pinot]
Posted by "ksnijjer (via GitHub)" <gi...@apache.org>.
ksnijjer commented on issue #12207:
URL: https://github.com/apache/pinot/issues/12207#issuecomment-1899148840
Thanks Gonzalo, I was verifying it differently - by looking at index_map and forward index size for the column (without MV encoding it's really huge). Not sure why you can't repro perhaps difference could be quick start using latest version of master vs original env using earlier.
Anyways if we can't repro with latest code then I am good to close unless @Jackie-Jiang has any additional thoughts/thinks to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] MV encoding config not working [pinot]
Posted by "ksnijjer (via GitHub)" <gi...@apache.org>.
ksnijjer commented on issue #12207:
URL: https://github.com/apache/pinot/issues/12207#issuecomment-1900989109
@gortiz will DM you the exact release, since it's StarTree release not sure how to map it to OSS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] MV encoding config not working [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #12207:
URL: https://github.com/apache/pinot/issues/12207#issuecomment-1898581172
I wasn't able to reproduce the issue. This is what I did:
1. Open `pinot-tools/src/main/resources/examples/batch/airlineStats/airlineStats_offline_table_config.json` and modified the `fieldConfigList` to add
``` json
{
"name": "DivAirportIDs",
"encodingType": "DICTIONARY",
"compressionCodec": "MV_ENTRY_DICT"
}
```
2. Started `pinot-tools/src/main/java/org/apache/pinot/tools/GenericQuickstart.java`
3. Checked that `airlineStats.DivAirportIDs` was using `MV_ENTRY_DICT`. I wasn't sure how to do this, so what I did was to add a breakpoint in both:
- `MultiValueFixedByteRawIndexCreator.java` constructor
- and `FixedBitMVEntryDictForwardIndexReader.java` constructor
4. Stopped the instance and delete temp data.
5. Modified `airlineStats_offline_table_config.json` again to change the `fieldConfigList` to:
```json
{
"name": "DivAirportIDs",
"encodingType": "DICTIONARY",
"indexes": {
"forward": {
"dictIdCompressionType": "MV_ENTRY_DICT"
}
}
}
```
6. Started `GenericQuickstart` again.
7. Checked that `airlineStats.DivAirportIDs` was using `MV_ENTRY_DICT` doing the same as before.
In both cases it looks like MV_ENTRY_DICT is being used. Am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] MV encoding config not working [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #12207:
URL: https://github.com/apache/pinot/issues/12207#issuecomment-1900772218
Before closing it, let me try with your version. Do you know which commit or tag was used so I can use the same one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] MV encoding config not working [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #12207: MV encoding config not working
URL: https://github.com/apache/pinot/issues/12207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] MV encoding config not working [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12207:
URL: https://github.com/apache/pinot/issues/12207#issuecomment-1897516652
@gortiz Can you help take a look? Not sure if I missed something
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] MV encoding config not working [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12207:
URL: https://github.com/apache/pinot/issues/12207#issuecomment-1958449473
@gortiz @ksnijjer Have you guys figured out the root cause?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org