You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2024/01/04 09:49:20 UTC

[PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]

gortiz opened a new pull request, #12218:
URL: https://github.com/apache/pinot/pull/12218

   This is a continuation on top of https://github.com/apache/pinot/pull/11993, which added a new configuration for forward indexes called `dictIdCompressionType`. In order to have more coherence, this PR adds the possibility of configuring that using a new literal `compressionCodec`. Valid values for this new field are:
   - Any valid value for dictIdCompressionType
   - Any valid value for chunkCompressionType
   
   This new field should be the default used to configure the forward index, given that it is coherent with the older syntax configuration and it is easier to remember.
   
   Feel free to discuss about other approaches.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12218:
URL: https://github.com/apache/pinot/pull/12218#discussion_r1457039663


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -99,6 +171,8 @@ public int hashCode() {
   }
 
   public static class Builder {
+    @Nullable
+    private CompressionCodec _compressionCodec;

Review Comment:
   I didn't want to remove them to keep backward compatibility, but I guess we didn't actually release a stable version with `chunkCompressionType` `dictIdCompressionType` so we may not need to worry about that.
   
   Thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12218:
URL: https://github.com/apache/pinot/pull/12218#issuecomment-1876855007

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: `30 lines` in your changes are missing coverage. Please review.
   > Comparison is base [(`d1cc17c`)](https://app.codecov.io/gh/apache/pinot/commit/d1cc17c5792b6284b29f7ed05a3dea01ac27ca29?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.53% compared to head [(`50516da`)](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 46.42%.
   
   | [Files](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...he/pinot/segment/spi/index/ForwardIndexConfig.java](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L0ZvcndhcmRJbmRleENvbmZpZy5qYXZh) | 33.33% | [26 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #12218       +/-   ##
   =============================================
   - Coverage     61.53%   46.42%   -15.11%     
   - Complexity      207      945      +738     
   =============================================
     Files          2416     1810      -606     
     Lines        131177    95209    -35968     
     Branches      20245    15359     -4886     
   =============================================
   - Hits          80717    44204    -36513     
   - Misses        44570    47848     +3278     
   + Partials       5890     3157     -2733     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.42% <33.33%> (-14.97%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.42% <33.33%> (-14.95%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.42% <33.33%> (-15.11%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.42% <33.33%> (-15.10%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.42% <33.33%> (-0.20%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12218/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12218?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #12218:
URL: https://github.com/apache/pinot/pull/12218#issuecomment-1897591980

   Agree we should use `compressionCodec` as the config key. The reason why I added `dictIdCompressionType` is to differentiate it from chunk based compression.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #12218:
URL: https://github.com/apache/pinot/pull/12218#issuecomment-1891860859

   > i am a bit confuse regarding this PR. so the goal is to use a single CompressionCodec to generate a valid combination of dict and chunk compression type yes?
   
   Yes. I think the approach in  #11993 is not correct. In I was telling @saurabhd336 that the approach in this PR may help him with the things he is working on right now related to log encoding.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12218:
URL: https://github.com/apache/pinot/pull/12218#discussion_r1457921930


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -99,6 +171,8 @@ public int hashCode() {
   }
 
   public static class Builder {
+    @Nullable
+    private CompressionCodec _compressionCodec;

Review Comment:
   IMO we don't need to keep `Builder` backward compatible. We do want to keep the config backward compatible. We may remove `dictIdCompressionType` from the config since that is not released yet, but should keep `chunkCompressionType`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12218:
URL: https://github.com/apache/pinot/pull/12218#discussion_r1462968861


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -99,6 +171,8 @@ public int hashCode() {
   }
 
   public static class Builder {
+    @Nullable
+    private CompressionCodec _compressionCodec;

Review Comment:
   Changed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #12218:
URL: https://github.com/apache/pinot/pull/12218


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #12218:
URL: https://github.com/apache/pinot/pull/12218#discussion_r1456698437


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -99,6 +171,8 @@ public int hashCode() {
   }
 
   public static class Builder {
+    @Nullable
+    private CompressionCodec _compressionCodec;

Review Comment:
   Since we can always tell `_chunkCompressionType` and `_dictIdCompressionType` from `_compressionCodec`, should we remove the other 2 fields?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Add the possibility of configuring ForwardIndexes with compressionCodec [pinot]

Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on PR #12218:
URL: https://github.com/apache/pinot/pull/12218#issuecomment-1894388950

   @saurabhd336 please share your thoughts :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org