You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2024/01/04 09:49:20 UTC
[PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]
gortiz opened a new pull request, #12218:
URL: https://github.com/apache/pinot/pull/12218
This is a continuation on top of https://github.com/apache/pinot/pull/11993, which added a new configuration for forward indexes called `dictIdCompressionType`. In order to have more coherence, this PR adds the possibility of configuring that using a new literal `compressionCodec`. Valid values for this new field are:
- Any valid value for dictIdCompressionType
- Any valid value for chunkCompressionType
This new field should be the default used to configure the forward index, given that it is coherent with the older syntax configuration and it is easier to remember.
Feel free to discuss about other approaches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12218:
URL: https://github.com/apache/pinot/pull/12218#discussion_r1457039663
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -99,6 +171,8 @@ public int hashCode() {
}
public static class Builder {
+ @Nullable
+ private CompressionCodec _compressionCodec;
Review Comment:
I didn't want to remove them to keep backward compatibility, but I guess we didn't actually release a stable version with `chunkCompressionType` `dictIdCompressionType` so we may not need to worry about that.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12218:
URL: https://github.com/apache/pinot/pull/12218#issuecomment-1876855007
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `30 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`d1cc17c`)](https://app.codecov.io/gh/apache/pinot/commit/d1cc17c5792b6284b29f7ed05a3dea01ac27ca29?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.53% compared to head [(`50516da`)](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 46.42%.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...he/pinot/segment/spi/index/ForwardIndexConfig.java](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L0ZvcndhcmRJbmRleENvbmZpZy5qYXZh) | 33.33% | [26 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #12218 +/- ##
=============================================
- Coverage 61.53% 46.42% -15.11%
- Complexity 207 945 +738
=============================================
Files 2416 1810 -606
Lines 131177 95209 -35968
Branches 20245 15359 -4886
=============================================
- Hits 80717 44204 -36513
- Misses 44570 47848 +3278
+ Partials 5890 3157 -2733
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.42% <33.33%> (-14.97%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.42% <33.33%> (-14.95%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.42% <33.33%> (-15.11%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.42% <33.33%> (-15.10%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.42% <33.33%> (-0.20%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #12218:
URL: https://github.com/apache/pinot/pull/12218#issuecomment-1897591980
Agree we should use `compressionCodec` as the config key. The reason why I added `dictIdCompressionType` is to differentiate it from chunk based compression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #12218:
URL: https://github.com/apache/pinot/pull/12218#issuecomment-1891860859
> i am a bit confuse regarding this PR. so the goal is to use a single CompressionCodec to generate a valid combination of dict and chunk compression type yes?
Yes. I think the approach in #11993 is not correct. In I was telling @saurabhd336 that the approach in this PR may help him with the things he is working on right now related to log encoding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12218:
URL: https://github.com/apache/pinot/pull/12218#discussion_r1457921930
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -99,6 +171,8 @@ public int hashCode() {
}
public static class Builder {
+ @Nullable
+ private CompressionCodec _compressionCodec;
Review Comment:
IMO we don't need to keep `Builder` backward compatible. We do want to keep the config backward compatible. We may remove `dictIdCompressionType` from the config since that is not released yet, but should keep `chunkCompressionType`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12218:
URL: https://github.com/apache/pinot/pull/12218#discussion_r1462968861
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -99,6 +171,8 @@ public int hashCode() {
}
public static class Builder {
+ @Nullable
+ private CompressionCodec _compressionCodec;
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #12218:
URL: https://github.com/apache/pinot/pull/12218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12218:
URL: https://github.com/apache/pinot/pull/12218#discussion_r1456698437
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -99,6 +171,8 @@ public int hashCode() {
}
public static class Builder {
+ @Nullable
+ private CompressionCodec _compressionCodec;
Review Comment:
Since we can always tell `_chunkCompressionType` and `_dictIdCompressionType` from `_compressionCodec`, should we remove the other 2 fields?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on PR #12218:
URL: https://github.com/apache/pinot/pull/12218#issuecomment-1894388950
@saurabhd336 please share your thoughts :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org