You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/02/21 03:09:44 UTC
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5086:
Remove the support of the old Star-Tree
Jackie-Jiang opened a new pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086
Remove the support of the old Star-Tree generated with the StarTreeIndexSpec (already deprecated)
Remove the related metadata (rawDoc and aggregatedDoc)
Remove the field size and derived metric type from MetricFieldSpec
Simplify the segment generation logic
Backward-incompatible: (checked with the community to make sure there is no use case with the old Star-Tree)
Existing segments with old Star-Tree won't work properly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull
request #5086: Remove the support of the old Star-Tree
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086#discussion_r384837537
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/stats/StringColumnPreIndexStatsCollector.java
##########
@@ -26,124 +26,82 @@
public class StringColumnPreIndexStatsCollector extends AbstractColumnStatisticsCollector {
- private String min;
- private String max;
+ private final ObjectSet<String> _values = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
- private int smallestStringLength = Integer.MAX_VALUE;
- private int longestStringLength = 0;
- private final ObjectSet<String> rawStringSet;
- private final ObjectSet<String> aggregatedStringSet;
- private String[] sortedStringList;
- private boolean sealed = false;
+ private int _minLength = Integer.MAX_VALUE;
+ private int _maxLength = 0;
+ private String[] _sortedValues;
+ private boolean _sealed = false;
public StringColumnPreIndexStatsCollector(String column, StatsCollectorConfig statsCollectorConfig) {
super(column, statsCollectorConfig);
- rawStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
- aggregatedStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
}
- /**
- * Collect statistics for the given entry.
- * - Add it to the passed in set (which could be raw or aggregated)
- * - Update maximum number of values for Multi-valued entries
- * - Update Total number of entries
- * - Check if entry is sorted.
- * @param entry
- * @param set
- */
- private void collectEntry(Object entry, ObjectSet<String> set) {
-
+ @Override
+ public void collect(Object entry) {
if (entry instanceof Object[]) {
- for (final Object e : (Object[]) entry) {
- String value = e.toString();
- set.add(value);
-
- int valueLength = StringUtil.encodeUtf8(value).length;
- smallestStringLength = Math.min(smallestStringLength, valueLength);
- longestStringLength = Math.max(longestStringLength, valueLength);
- }
- if (maxNumberOfMultiValues < ((Object[]) entry).length) {
- maxNumberOfMultiValues = ((Object[]) entry).length;
+ Object[] values = (Object[]) entry;
+ for (Object obj : values) {
+ String value = (String) obj;
+ _values.add(value);
+
+ int length = StringUtil.encodeUtf8(value).length;
+ _minLength = Math.min(_minLength, length);
+ _maxLength = Math.max(_maxLength, length);
}
- updateTotalNumberOfEntries((Object[]) entry);
- } else {
- String value;
- if (entry != null) {
- value = entry.toString();
- } else {
- value = fieldSpec.getDefaultNullValue().toString();
Review comment:
Null values are handled by NullValueTransformer. Here we will never get null entries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io commented on issue #5086: Remove the
support of the old Star-Tree
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086#issuecomment-589492764
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=h1) Report
> Merging [#5086](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/35f21174075af3318f1b6dd59ec428579fc24a43?src=pr&el=desc) will **decrease** coverage by `0.43%`.
> The diff coverage is `79.19%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #5086 +/- ##
============================================
- Coverage 58.26% 57.82% -0.44%
Complexity 12 12
============================================
Files 1209 1194 -15
Lines 64529 63035 -1494
Branches 9483 9292 -191
============================================
- Hits 37595 36450 -1145
+ Misses 24174 23873 -301
+ Partials 2760 2712 -48
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...e/pinot/core/segment/creator/impl/V1Constants.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2NyZWF0b3IvaW1wbC9WMUNvbnN0YW50cy5qYXZh) | `12.5% <ø> (+1.38%)` | `0 <0> (ø)` | :arrow_down: |
| [...ot/core/startree/v2/store/StarTreeLoaderUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS92Mi9zdG9yZS9TdGFyVHJlZUxvYWRlclV0aWxzLmphdmE=) | `94.73% <ø> (+1.08%)` | `0 <0> (ø)` | :arrow_down: |
| [...main/java/org/apache/pinot/spi/data/FieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9GaWVsZFNwZWMuamF2YQ==) | `85.82% <ø> (-0.12%)` | `0 <0> (ø)` | |
| [.../impl/stats/AbstractColumnStatisticsCollector.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2NyZWF0b3IvaW1wbC9zdGF0cy9BYnN0cmFjdENvbHVtblN0YXRpc3RpY3NDb2xsZWN0b3IuamF2YQ==) | `94.28% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
| [...va/org/apache/pinot/core/minion/SegmentPurger.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vU2VnbWVudFB1cmdlci5qYXZh) | `78.88% <ø> (+1.46%)` | `0 <0> (ø)` | :arrow_down: |
| [...org/apache/pinot/common/config/IndexingConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL0luZGV4aW5nQ29uZmlnLmphdmE=) | `100% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
| [...pinot/core/segment/store/ColumnIndexDirectory.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L3N0b3JlL0NvbHVtbkluZGV4RGlyZWN0b3J5LmphdmE=) | `100% <ø> (+6.66%)` | `0 <0> (ø)` | :arrow_down: |
| [...ndex/converter/SegmentV1V2ToV3FormatConverter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2luZGV4L2NvbnZlcnRlci9TZWdtZW50VjFWMlRvVjNGb3JtYXRDb252ZXJ0ZXIuamF2YQ==) | `75.93% <ø> (-1.21%)` | `0 <0> (ø)` | |
| [...ot/core/segment/store/SegmentLocalFSDirectory.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L3N0b3JlL1NlZ21lbnRMb2NhbEZTRGlyZWN0b3J5LmphdmE=) | `68.12% <ø> (+1.82%)` | `0 <0> (ø)` | :arrow_down: |
| [...inot/tools/admin/command/CreateSegmentCommand.java](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree#diff-cGlub3QtdG9vbHMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3Rvb2xzL2FkbWluL2NvbW1hbmQvQ3JlYXRlU2VnbWVudENvbW1hbmQuamF2YQ==) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
| ... and [133 more](https://codecov.io/gh/apache/incubator-pinot/pull/5086/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=footer). Last update [35f2117...51abe2e](https://codecov.io/gh/apache/incubator-pinot/pull/5086?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #5086: Remove
the support of the old Star-Tree
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull
request #5086: Remove the support of the old Star-Tree
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086#discussion_r384837002
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/stats/StringColumnPreIndexStatsCollector.java
##########
@@ -26,124 +26,82 @@
public class StringColumnPreIndexStatsCollector extends AbstractColumnStatisticsCollector {
- private String min;
- private String max;
+ private final ObjectSet<String> _values = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
- private int smallestStringLength = Integer.MAX_VALUE;
- private int longestStringLength = 0;
- private final ObjectSet<String> rawStringSet;
- private final ObjectSet<String> aggregatedStringSet;
- private String[] sortedStringList;
- private boolean sealed = false;
+ private int _minLength = Integer.MAX_VALUE;
+ private int _maxLength = 0;
+ private String[] _sortedValues;
+ private boolean _sealed = false;
public StringColumnPreIndexStatsCollector(String column, StatsCollectorConfig statsCollectorConfig) {
super(column, statsCollectorConfig);
- rawStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
- aggregatedStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
}
- /**
- * Collect statistics for the given entry.
- * - Add it to the passed in set (which could be raw or aggregated)
- * - Update maximum number of values for Multi-valued entries
- * - Update Total number of entries
- * - Check if entry is sorted.
- * @param entry
- * @param set
- */
- private void collectEntry(Object entry, ObjectSet<String> set) {
-
+ @Override
+ public void collect(Object entry) {
if (entry instanceof Object[]) {
- for (final Object e : (Object[]) entry) {
- String value = e.toString();
- set.add(value);
-
- int valueLength = StringUtil.encodeUtf8(value).length;
- smallestStringLength = Math.min(smallestStringLength, valueLength);
- longestStringLength = Math.max(longestStringLength, valueLength);
- }
- if (maxNumberOfMultiValues < ((Object[]) entry).length) {
- maxNumberOfMultiValues = ((Object[]) entry).length;
+ Object[] values = (Object[]) entry;
+ for (Object obj : values) {
+ String value = (String) obj;
+ _values.add(value);
+
+ int length = StringUtil.encodeUtf8(value).length;
+ _minLength = Math.min(_minLength, length);
+ _maxLength = Math.max(_maxLength, length);
}
- updateTotalNumberOfEntries((Object[]) entry);
- } else {
- String value;
- if (entry != null) {
- value = entry.toString();
- } else {
- value = fieldSpec.getDefaultNullValue().toString();
Review comment:
Ignore. Got it.... this is STRING column
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull
request #5086: Remove the support of the old Star-Tree
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086#discussion_r384836846
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/stats/StringColumnPreIndexStatsCollector.java
##########
@@ -26,124 +26,82 @@
public class StringColumnPreIndexStatsCollector extends AbstractColumnStatisticsCollector {
- private String min;
- private String max;
+ private final ObjectSet<String> _values = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
- private int smallestStringLength = Integer.MAX_VALUE;
- private int longestStringLength = 0;
- private final ObjectSet<String> rawStringSet;
- private final ObjectSet<String> aggregatedStringSet;
- private String[] sortedStringList;
- private boolean sealed = false;
+ private int _minLength = Integer.MAX_VALUE;
+ private int _maxLength = 0;
+ private String[] _sortedValues;
+ private boolean _sealed = false;
public StringColumnPreIndexStatsCollector(String column, StatsCollectorConfig statsCollectorConfig) {
super(column, statsCollectorConfig);
- rawStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
- aggregatedStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
}
- /**
- * Collect statistics for the given entry.
- * - Add it to the passed in set (which could be raw or aggregated)
- * - Update maximum number of values for Multi-valued entries
- * - Update Total number of entries
- * - Check if entry is sorted.
- * @param entry
- * @param set
- */
- private void collectEntry(Object entry, ObjectSet<String> set) {
-
+ @Override
+ public void collect(Object entry) {
if (entry instanceof Object[]) {
- for (final Object e : (Object[]) entry) {
- String value = e.toString();
- set.add(value);
-
- int valueLength = StringUtil.encodeUtf8(value).length;
- smallestStringLength = Math.min(smallestStringLength, valueLength);
- longestStringLength = Math.max(longestStringLength, valueLength);
- }
- if (maxNumberOfMultiValues < ((Object[]) entry).length) {
- maxNumberOfMultiValues = ((Object[]) entry).length;
+ Object[] values = (Object[]) entry;
+ for (Object obj : values) {
+ String value = (String) obj;
+ _values.add(value);
+
+ int length = StringUtil.encodeUtf8(value).length;
+ _minLength = Math.min(_minLength, length);
+ _maxLength = Math.max(_maxLength, length);
}
- updateTotalNumberOfEntries((Object[]) entry);
- } else {
- String value;
- if (entry != null) {
- value = entry.toString();
- } else {
- value = fieldSpec.getDefaultNullValue().toString();
Review comment:
Why this change of not using getDefaultNullValue()?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull
request #5086: Remove the support of the old Star-Tree
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5086: Remove the support of the old Star-Tree
URL: https://github.com/apache/incubator-pinot/pull/5086#discussion_r384837002
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/stats/StringColumnPreIndexStatsCollector.java
##########
@@ -26,124 +26,82 @@
public class StringColumnPreIndexStatsCollector extends AbstractColumnStatisticsCollector {
- private String min;
- private String max;
+ private final ObjectSet<String> _values = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
- private int smallestStringLength = Integer.MAX_VALUE;
- private int longestStringLength = 0;
- private final ObjectSet<String> rawStringSet;
- private final ObjectSet<String> aggregatedStringSet;
- private String[] sortedStringList;
- private boolean sealed = false;
+ private int _minLength = Integer.MAX_VALUE;
+ private int _maxLength = 0;
+ private String[] _sortedValues;
+ private boolean _sealed = false;
public StringColumnPreIndexStatsCollector(String column, StatsCollectorConfig statsCollectorConfig) {
super(column, statsCollectorConfig);
- rawStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
- aggregatedStringSet = new ObjectOpenHashSet<>(INITIAL_HASH_SET_SIZE);
}
- /**
- * Collect statistics for the given entry.
- * - Add it to the passed in set (which could be raw or aggregated)
- * - Update maximum number of values for Multi-valued entries
- * - Update Total number of entries
- * - Check if entry is sorted.
- * @param entry
- * @param set
- */
- private void collectEntry(Object entry, ObjectSet<String> set) {
-
+ @Override
+ public void collect(Object entry) {
if (entry instanceof Object[]) {
- for (final Object e : (Object[]) entry) {
- String value = e.toString();
- set.add(value);
-
- int valueLength = StringUtil.encodeUtf8(value).length;
- smallestStringLength = Math.min(smallestStringLength, valueLength);
- longestStringLength = Math.max(longestStringLength, valueLength);
- }
- if (maxNumberOfMultiValues < ((Object[]) entry).length) {
- maxNumberOfMultiValues = ((Object[]) entry).length;
+ Object[] values = (Object[]) entry;
+ for (Object obj : values) {
+ String value = (String) obj;
+ _values.add(value);
+
+ int length = StringUtil.encodeUtf8(value).length;
+ _minLength = Math.min(_minLength, length);
+ _maxLength = Math.max(_maxLength, length);
}
- updateTotalNumberOfEntries((Object[]) entry);
- } else {
- String value;
- if (entry != null) {
- value = entry.toString();
- } else {
- value = fieldSpec.getDefaultNullValue().toString();
Review comment:
Ignore. Got it.... this is STRING column
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org