You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/02/21 03:09:44 UTC

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5086: Remove the support of the old Star-Tree

Jackie-Jiang opened a new pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086
 
 
   Remove the support of the old Star-Tree generated with the StarTreeIndexSpec (already deprecated)
   Remove the related metadata (rawDoc and aggregatedDoc)
   Remove the field size and derived metric type from MetricFieldSpec
   Simplify the segment generation logic
   
   Backward-incompatible: (checked with the community to make sure there is no use case with the old Star-Tree)
   Existing segments with old Star-Tree won't work properly

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5086: Remove the support of the old Star-Tree

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086#discussion_r384837537
 
 

 ##########
 File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/stats/StringColumnPreIndexStatsCollector.java
 ##########
 @@ -26,124 +26,82 @@
 
 
 public class StringColumnPreIndexStatsCollector extends AbstractColumnStatisticsCollector {
-  private String min;
-  private String max;
+  private final ObjectSet<String> _values = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
 
-  private int smallestStringLength = Integer.MAX_VALUE;
-  private int longestStringLength = 0;
-  private final ObjectSet<String> rawStringSet;
-  private final ObjectSet<String> aggregatedStringSet;
-  private String[] sortedStringList;
-  private boolean sealed = false;
+  private int _minLength = Integer.MAX_VALUE;
+  private int _maxLength = 0;
+  private String[] _sortedValues;
+  private boolean _sealed = false;
 
   public StringColumnPreIndexStatsCollector(String column, StatsCollectorConfig statsCollectorConfig) {
     super(column, statsCollectorConfig);
-    rawStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
-    aggregatedStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
   }
 
-  /**
-   * Collect statistics for the given entry.
-   * - Add it to the passed in set (which could be raw or aggregated)
-   * - Update maximum number of values for Multi-valued entries
-   * - Update Total number of entries
-   * - Check if entry is sorted.
-   * @param entry
-   * @param set
-   */
-  private void collectEntry(Object entry, ObjectSet<String> set) {
-
+  @Override
+  public void collect(Object entry) {
     if (entry instanceof Object[]) {
-      for (final Object e : (Object[]) entry) {
-        String value = e.toString();
-        set.add(value);
-
-        int valueLength = StringUtil.encodeUtf8(value).length;
-        smallestStringLength = Math.min(smallestStringLength, valueLength);
-        longestStringLength = Math.max(longestStringLength, valueLength);
-      }
-      if (maxNumberOfMultiValues < ((Object[]) entry).length) {
-        maxNumberOfMultiValues = ((Object[]) entry).length;
+      Object[] values = (Object[]) entry;
+      for (Object obj : values) {
+        String value = (String) obj;
+        _values.add(value);
+
+        int length = StringUtil.encodeUtf8(value).length;
+        _minLength = Math.min(_minLength, length);
+        _maxLength = Math.max(_maxLength, length);
       }
-      updateTotalNumberOfEntries((Object[]) entry);
-    } else {
 
-      String value;
-      if (entry != null) {
-        value = entry.toString();
-      } else {
-        value = fieldSpec.getDefaultNullValue().toString();
 
 Review comment:
   Null values are handled by NullValueTransformer. Here we will never get null entries.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-io commented on issue #5086: Remove the support of the old Star-Tree

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086#issuecomment-589492764
 
 
   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=h1) Report
   > Merging [#5086](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/35f21174075af3318f1b6dd59ec428579fc24a43?src=pr&el=desc) will **decrease** coverage by `0.43%`.
   > The diff coverage is `79.19%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/5086/graphs/tree.svg?width=650&token=4ibza2ugkz&height=150&src=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #5086      +/-   ##
   ============================================
   - Coverage     58.26%   57.82%   -0.44%     
     Complexity       12       12              
   ============================================
     Files          1209     1194      -15     
     Lines         64529    63035    -1494     
     Branches       9483     9292     -191     
   ============================================
   - Hits          37595    36450    -1145     
   + Misses        24174    23873     -301     
   + Partials       2760     2712      -48
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...e/pinot/core/segment/creator/impl/V1Constants.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2NyZWF0b3IvaW1wbC9WMUNvbnN0YW50cy5qYXZh) | `12.5% <ø> (+1.38%)` | `0 <0> (ø)` | :arrow_down: |
   | [...ot/core/startree/v2/store/StarTreeLoaderUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS92Mi9zdG9yZS9TdGFyVHJlZUxvYWRlclV0aWxzLmphdmE=) | `94.73% <ø> (+1.08%)` | `0 <0> (ø)` | :arrow_down: |
   | [...main/java/org/apache/pinot/spi/data/FieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9GaWVsZFNwZWMuamF2YQ==) | `85.82% <ø> (-0.12%)` | `0 <0> (ø)` | |
   | [.../impl/stats/AbstractColumnStatisticsCollector.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2NyZWF0b3IvaW1wbC9zdGF0cy9BYnN0cmFjdENvbHVtblN0YXRpc3RpY3NDb2xsZWN0b3IuamF2YQ==) | `94.28% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...va/org/apache/pinot/core/minion/SegmentPurger.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vU2VnbWVudFB1cmdlci5qYXZh) | `78.88% <ø> (+1.46%)` | `0 <0> (ø)` | :arrow_down: |
   | [...org/apache/pinot/common/config/IndexingConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL0luZGV4aW5nQ29uZmlnLmphdmE=) | `100% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [...pinot/core/segment/store/ColumnIndexDirectory.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L3N0b3JlL0NvbHVtbkluZGV4RGlyZWN0b3J5LmphdmE=) | `100% <ø> (+6.66%)` | `0 <0> (ø)` | :arrow_down: |
   | [...ndex/converter/SegmentV1V2ToV3FormatConverter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2luZGV4L2NvbnZlcnRlci9TZWdtZW50VjFWMlRvVjNGb3JtYXRDb252ZXJ0ZXIuamF2YQ==) | `75.93% <ø> (-1.21%)` | `0 <0> (ø)` | |
   | [...ot/core/segment/store/SegmentLocalFSDirectory.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L3N0b3JlL1NlZ21lbnRMb2NhbEZTRGlyZWN0b3J5LmphdmE=) | `68.12% <ø> (+1.82%)` | `0 <0> (ø)` | :arrow_down: |
   | [...inot/tools/admin/command/CreateSegmentCommand.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtdG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3Rvb2xzL2FkbWluL2NvbW1hbmQvQ3JlYXRlU2VnbWVudENvbW1hbmQuamF2YQ==) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | ... and [133 more](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=footer). Last update [35f2117...51abe2e](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #5086: Remove the support of the old Star-Tree

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5086: Remove the support of the old Star-Tree

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086#discussion_r384837002
 
 

 ##########
 File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/stats/StringColumnPreIndexStatsCollector.java
 ##########
 @@ -26,124 +26,82 @@
 
 
 public class StringColumnPreIndexStatsCollector extends AbstractColumnStatisticsCollector {
-  private String min;
-  private String max;
+  private final ObjectSet<String> _values = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
 
-  private int smallestStringLength = Integer.MAX_VALUE;
-  private int longestStringLength = 0;
-  private final ObjectSet<String> rawStringSet;
-  private final ObjectSet<String> aggregatedStringSet;
-  private String[] sortedStringList;
-  private boolean sealed = false;
+  private int _minLength = Integer.MAX_VALUE;
+  private int _maxLength = 0;
+  private String[] _sortedValues;
+  private boolean _sealed = false;
 
   public StringColumnPreIndexStatsCollector(String column, StatsCollectorConfig statsCollectorConfig) {
     super(column, statsCollectorConfig);
-    rawStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
-    aggregatedStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
   }
 
-  /**
-   * Collect statistics for the given entry.
-   * - Add it to the passed in set (which could be raw or aggregated)
-   * - Update maximum number of values for Multi-valued entries
-   * - Update Total number of entries
-   * - Check if entry is sorted.
-   * @param entry
-   * @param set
-   */
-  private void collectEntry(Object entry, ObjectSet<String> set) {
-
+  @Override
+  public void collect(Object entry) {
     if (entry instanceof Object[]) {
-      for (final Object e : (Object[]) entry) {
-        String value = e.toString();
-        set.add(value);
-
-        int valueLength = StringUtil.encodeUtf8(value).length;
-        smallestStringLength = Math.min(smallestStringLength, valueLength);
-        longestStringLength = Math.max(longestStringLength, valueLength);
-      }
-      if (maxNumberOfMultiValues < ((Object[]) entry).length) {
-        maxNumberOfMultiValues = ((Object[]) entry).length;
+      Object[] values = (Object[]) entry;
+      for (Object obj : values) {
+        String value = (String) obj;
+        _values.add(value);
+
+        int length = StringUtil.encodeUtf8(value).length;
+        _minLength = Math.min(_minLength, length);
+        _maxLength = Math.max(_maxLength, length);
       }
-      updateTotalNumberOfEntries((Object[]) entry);
-    } else {
 
-      String value;
-      if (entry != null) {
-        value = entry.toString();
-      } else {
-        value = fieldSpec.getDefaultNullValue().toString();
 
 Review comment:
   Ignore. Got it.... this is STRING column

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5086: Remove the support of the old Star-Tree

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086#discussion_r384836846
 
 

 ##########
 File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/stats/StringColumnPreIndexStatsCollector.java
 ##########
 @@ -26,124 +26,82 @@
 
 
 public class StringColumnPreIndexStatsCollector extends AbstractColumnStatisticsCollector {
-  private String min;
-  private String max;
+  private final ObjectSet<String> _values = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
 
-  private int smallestStringLength = Integer.MAX_VALUE;
-  private int longestStringLength = 0;
-  private final ObjectSet<String> rawStringSet;
-  private final ObjectSet<String> aggregatedStringSet;
-  private String[] sortedStringList;
-  private boolean sealed = false;
+  private int _minLength = Integer.MAX_VALUE;
+  private int _maxLength = 0;
+  private String[] _sortedValues;
+  private boolean _sealed = false;
 
   public StringColumnPreIndexStatsCollector(String column, StatsCollectorConfig statsCollectorConfig) {
     super(column, statsCollectorConfig);
-    rawStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
-    aggregatedStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
   }
 
-  /**
-   * Collect statistics for the given entry.
-   * - Add it to the passed in set (which could be raw or aggregated)
-   * - Update maximum number of values for Multi-valued entries
-   * - Update Total number of entries
-   * - Check if entry is sorted.
-   * @param entry
-   * @param set
-   */
-  private void collectEntry(Object entry, ObjectSet<String> set) {
-
+  @Override
+  public void collect(Object entry) {
     if (entry instanceof Object[]) {
-      for (final Object e : (Object[]) entry) {
-        String value = e.toString();
-        set.add(value);
-
-        int valueLength = StringUtil.encodeUtf8(value).length;
-        smallestStringLength = Math.min(smallestStringLength, valueLength);
-        longestStringLength = Math.max(longestStringLength, valueLength);
-      }
-      if (maxNumberOfMultiValues < ((Object[]) entry).length) {
-        maxNumberOfMultiValues = ((Object[]) entry).length;
+      Object[] values = (Object[]) entry;
+      for (Object obj : values) {
+        String value = (String) obj;
+        _values.add(value);
+
+        int length = StringUtil.encodeUtf8(value).length;
+        _minLength = Math.min(_minLength, length);
+        _maxLength = Math.max(_maxLength, length);
       }
-      updateTotalNumberOfEntries((Object[]) entry);
-    } else {
 
-      String value;
-      if (entry != null) {
-        value = entry.toString();
-      } else {
-        value = fieldSpec.getDefaultNullValue().toString();
 
 Review comment:
   Why this change of not using getDefaultNullValue()?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5086: Remove the support of the old Star-Tree

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086#discussion_r384837002
 
 

 ##########
 File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/stats/StringColumnPreIndexStatsCollector.java
 ##########
 @@ -26,124 +26,82 @@
 
 
 public class StringColumnPreIndexStatsCollector extends AbstractColumnStatisticsCollector {
-  private String min;
-  private String max;
+  private final ObjectSet<String> _values = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
 
-  private int smallestStringLength = Integer.MAX_VALUE;
-  private int longestStringLength = 0;
-  private final ObjectSet<String> rawStringSet;
-  private final ObjectSet<String> aggregatedStringSet;
-  private String[] sortedStringList;
-  private boolean sealed = false;
+  private int _minLength = Integer.MAX_VALUE;
+  private int _maxLength = 0;
+  private String[] _sortedValues;
+  private boolean _sealed = false;
 
   public StringColumnPreIndexStatsCollector(String column, StatsCollectorConfig statsCollectorConfig) {
     super(column, statsCollectorConfig);
-    rawStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
-    aggregatedStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
   }
 
-  /**
-   * Collect statistics for the given entry.
-   * - Add it to the passed in set (which could be raw or aggregated)
-   * - Update maximum number of values for Multi-valued entries
-   * - Update Total number of entries
-   * - Check if entry is sorted.
-   * @param entry
-   * @param set
-   */
-  private void collectEntry(Object entry, ObjectSet<String> set) {
-
+  @Override
+  public void collect(Object entry) {
     if (entry instanceof Object[]) {
-      for (final Object e : (Object[]) entry) {
-        String value = e.toString();
-        set.add(value);
-
-        int valueLength = StringUtil.encodeUtf8(value).length;
-        smallestStringLength = Math.min(smallestStringLength, valueLength);
-        longestStringLength = Math.max(longestStringLength, valueLength);
-      }
-      if (maxNumberOfMultiValues < ((Object[]) entry).length) {
-        maxNumberOfMultiValues = ((Object[]) entry).length;
+      Object[] values = (Object[]) entry;
+      for (Object obj : values) {
+        String value = (String) obj;
+        _values.add(value);
+
+        int length = StringUtil.encodeUtf8(value).length;
+        _minLength = Math.min(_minLength, length);
+        _maxLength = Math.max(_maxLength, length);
       }
-      updateTotalNumberOfEntries((Object[]) entry);
-    } else {
 
-      String value;
-      if (entry != null) {
-        value = entry.toString();
-      } else {
-        value = fieldSpec.getDefaultNullValue().toString();
 
 Review comment:
   Ignore. Got it.... this is STRING column

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org