You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/05/02 19:23:22 UTC
[GitHub] [incubator-pinot] kriti-sc opened a new pull request #6873: feature/#6766 JSON and Startree index information in API
kriti-sc opened a new pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r633055126
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -62,24 +76,63 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
columnSet = new HashSet<>(columns);
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
- segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(COLUMN_INDEX_KEY, JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
Review comment:
This is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#issuecomment-840887487
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6873](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2f615b5) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/d5e7106546f0b134a29028b2065bba28929d5e8e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d5e7106) will **decrease** coverage by `8.62%`.
> The diff coverage is `21.42%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6873 +/- ##
============================================
- Coverage 74.05% 65.42% -8.63%
Complexity 12 12
============================================
Files 1421 1432 +11
Lines 69141 70621 +1480
Branches 9986 10209 +223
============================================
- Hits 51204 46206 -4998
- Misses 14590 21112 +6522
+ Partials 3347 3303 -44
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| integration | `?` | `?` | |
| unittests | `65.42% <21.42%> (-0.53%)` | `12.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...t/server/api/resources/SegmentMetadataFetcher.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvYXBpL3Jlc291cmNlcy9TZWdtZW50TWV0YWRhdGFGZXRjaGVyLmphdmE=) | `45.00% <21.42%> (-24.05%)` | `0.00 <0.00> (ø)` | |
| [...a/org/apache/pinot/minion/metrics/MinionMeter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../apache/pinot/minion/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../core/startree/plan/StarTreeTransformPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlVHJhbnNmb3JtUGxhbk5vZGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...core/startree/plan/StarTreeProjectionPlanNode.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlUHJvamVjdGlvblBsYW5Ob2RlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...t/minion/executor/MinionTaskZkMetadataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhlY3V0b3IvTWluaW9uVGFza1prTWV0YWRhdGFNYW5hZ2VyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| ... and [494 more](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d5e7106...2f615b5](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r633055126
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -62,24 +76,63 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
columnSet = new HashSet<>(columns);
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
- segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(COLUMN_INDEX_KEY, JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
Review comment:
This is done.
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -62,24 +76,63 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
columnSet = new HashSet<>(columns);
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
- segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(COLUMN_INDEX_KEY, JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+ ImmutableSegmentDataManager immutableSegmentDataManager = (ImmutableSegmentDataManager) segmentDataManager;
+ ImmutableSegment immutableSegment = immutableSegmentDataManager.getSegment();
+ List<StarTreeV2> starTrees = immutableSegment.getStarTrees();
+ if (starTrees == null) {
+ return startreeDetails;
+ }
+ startreeDetails = getImmutableSegmentStartreeIndexes(starTrees);
+ return startreeDetails;
Review comment:
This is done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r633950800
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -128,8 +137,49 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
indexStatus.put(RANGE_INDEX, INDEX_AVAILABLE);
}
+ if (Objects.isNull(columnIndexContainer.getJsonIndex())){
+ indexStatus.put(JSON_INDEX, INDEX_NOT_AVAILABLE);
+ } else {
+ indexStatus.put(JSON_INDEX, INDEX_AVAILABLE);
+ }
+
columnIndexMap.put(entry.getKey(), indexStatus);
}
return columnIndexMap;
}
+
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
Review comment:
```suggestion
@Nullable
private static List<Map<String, Object>> getStarTreeIndexes(SegmentDataManager segmentDataManager) {
```
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -62,25 +75,21 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
columnSet = new HashSet<>(columns);
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
- segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(COLUMN_INDEX_KEY, JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
/**
* Get the JSON object with the segment column's indexing metadata.
*/
private static Map<String, Map<String, String>> getIndexesForSegmentColumns(SegmentDataManager segmentDataManager) {
Review comment:
Annotate with nullable
##########
File path: pinot-server/src/test/java/org/apache/pinot/server/api/TablesResourceTest.java
##########
@@ -273,6 +273,7 @@ public void testOfflineTableSegmentMetadata()
.get(String.class));
Assert.assertEquals(jsonResponse.get("columns").size(), 2);
Assert.assertEquals(jsonResponse.get("indexes").size(), 17);
+ Assert.assertEquals(jsonResponse.get("star-tree-index").size(), 0);
Review comment:
You might need to modify the test because it can be `null` here
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -128,8 +137,49 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
indexStatus.put(RANGE_INDEX, INDEX_AVAILABLE);
}
+ if (Objects.isNull(columnIndexContainer.getJsonIndex())){
+ indexStatus.put(JSON_INDEX, INDEX_NOT_AVAILABLE);
+ } else {
+ indexStatus.put(JSON_INDEX, INDEX_AVAILABLE);
+ }
+
columnIndexMap.put(entry.getKey(), indexStatus);
}
return columnIndexMap;
}
+
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<StarTreeV2> starTrees = segmentDataManager.getSegment().getStarTrees();
+ return starTrees != null ? getImmutableSegmentStartreeIndexes(starTrees) : null;
+ }
+
+ /**
+ * Helper to loop over star trees of a segment to create a map containing star tree details.
+ */
+ private static List<Map<String, Object>> getImmutableSegmentStartreeIndexes(List<StarTreeV2> starTrees){
Review comment:
```suggestion
private static List<Map<String, Object>> getStarTreeIndexes(List<StarTreeV2> starTrees){
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter commented on pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#issuecomment-840887487
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6873](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1ab6007) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/d5e7106546f0b134a29028b2065bba28929d5e8e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d5e7106) will **decrease** coverage by `30.94%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6873 +/- ##
=============================================
- Coverage 74.05% 43.11% -30.95%
+ Complexity 12 7 -5
=============================================
Files 1421 1430 +9
Lines 69141 70568 +1427
Branches 9986 10202 +216
=============================================
- Hits 51204 30426 -20778
- Misses 14590 37514 +22924
+ Partials 3347 2628 -719
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| integration | `43.11% <0.00%> (-0.53%)` | `7.00 <0.00> (ø)` | |
| unittests | `?` | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...t/server/api/resources/SegmentMetadataFetcher.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvYXBpL3Jlc291cmNlcy9TZWdtZW50TWV0YWRhdGFGZXRjaGVyLmphdmE=) | `0.00% <0.00%> (-69.05%)` | `0.00 <0.00> (ø)` | |
| [...c/main/java/org/apache/pinot/common/tier/Tier.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...a/org/apache/pinot/spi/config/table/TableType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL1RhYmxlVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...a/org/apache/pinot/spi/data/DateTimeFieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9EYXRlVGltZUZpZWxkU3BlYy5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...a/org/apache/pinot/spi/env/PinotConfiguration.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZW52L1Bpbm90Q29uZmlndXJhdGlvbi5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../org/apache/pinot/spi/data/DimensionFieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9EaW1lbnNpb25GaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [.../org/apache/pinot/spi/data/readers/FileFormat.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9yZWFkZXJzL0ZpbGVGb3JtYXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...org/apache/pinot/spi/config/table/QuotaConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL1F1b3RhQ29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| [...org/apache/pinot/spi/config/tenant/TenantRole.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RlbmFudC9UZW5hbnRSb2xlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (ø%)` | |
| ... and [957 more](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d5e7106...1ab6007](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r632141130
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -62,24 +76,63 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
columnSet = new HashSet<>(columns);
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
- segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(COLUMN_INDEX_KEY, JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+ ImmutableSegmentDataManager immutableSegmentDataManager = (ImmutableSegmentDataManager) segmentDataManager;
+ ImmutableSegment immutableSegment = immutableSegmentDataManager.getSegment();
+ List<StarTreeV2> starTrees = immutableSegment.getStarTrees();
+ if (starTrees == null) {
+ return startreeDetails;
+ }
+ startreeDetails = getImmutableSegmentStartreeIndexes(starTrees);
+ return startreeDetails;
Review comment:
Can be simplified. Also suggest returning `null` if index does not exist
```suggestion
List<StarTreeV2> starTrees = segmentDataManager.getSegment().getStarTrees();
return starTrees != null ? getImmutableSegmentStartreeIndexes(starTrees) : null;
```
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -62,24 +76,63 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
columnSet = new HashSet<>(columns);
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
- segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(COLUMN_INDEX_KEY, JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+ ImmutableSegmentDataManager immutableSegmentDataManager = (ImmutableSegmentDataManager) segmentDataManager;
+ ImmutableSegment immutableSegment = immutableSegmentDataManager.getSegment();
+ List<StarTreeV2> starTrees = immutableSegment.getStarTrees();
+ if (starTrees == null) {
+ return startreeDetails;
+ }
+ startreeDetails = getImmutableSegmentStartreeIndexes(starTrees);
+ return startreeDetails;
+ }
+
+ /**
+ * Helper to loop over star trees of a segment to create a map containing star tree details.
+ */
+ private static List<Map<String, Object>> getImmutableSegmentStartreeIndexes(List<StarTreeV2> starTrees){
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+ for (StarTreeV2 starTree : starTrees) {
+ StarTreeV2Metadata starTreeMetadata = starTree.getMetadata();
+
+ Map<String, Object> starTreeIndexMap = new LinkedHashMap<>();
+
+ List<String> starTreeDimensions = starTreeMetadata.getDimensionsSplitOrder();
+ starTreeIndexMap.put(STAR_TREE_DIMENSION_COLUMNS, starTreeDimensions);
+
+ List<String> starTreeMetricAggregations = new ArrayList<>();
+ Set<AggregationFunctionColumnPair> functionColumnPairs = starTreeMetadata.getFunctionColumnPairs();
+ for (AggregationFunctionColumnPair functionColumnPair : functionColumnPairs) {
+ starTreeMetricAggregations.add(functionColumnPair.toColumnName());
+ }
+ starTreeIndexMap.put(STAR_TREE_METRIC_AGGREGATIONS, starTreeMetricAggregations);
+
+ starTreeIndexMap.put(STAR_TREE_MAX_LEAF_RECORDS, starTreeMetadata.getMaxLeafRecords());
+ starTreeIndexMap.put(STAR_TREE_DIMENSION_COLUMNS_SKIPPED, starTreeMetadata.getSkipStarNodeCreationForDimensions());
+ startreeDetails.add(starTreeIndexMap);
+ }
+ return startreeDetails;
+ }
+
/**
* Get the JSON object with the segment column's indexing metadata.
*/
private static Map<String, Map<String, String>> getIndexesForSegmentColumns(SegmentDataManager segmentDataManager) {
Map<String, Map<String, String>> columnIndexMap = null;
- if (segmentDataManager instanceof ImmutableSegmentDataManager) {
- ImmutableSegmentDataManager immutableSegmentDataManager = (ImmutableSegmentDataManager) segmentDataManager;
- ImmutableSegment immutableSegment = immutableSegmentDataManager.getSegment();
- if (immutableSegment instanceof ImmutableSegmentImpl) {
- ImmutableSegmentImpl immutableSegmentImpl = (ImmutableSegmentImpl) immutableSegment;
- Map<String, ColumnIndexContainer> columnIndexContainerMap = immutableSegmentImpl.getIndexContainerMap();
- columnIndexMap = getImmutableSegmentColumnIndexes(columnIndexContainerMap);
- }
- }
+ ImmutableSegmentDataManager immutableSegmentDataManager = (ImmutableSegmentDataManager) segmentDataManager;
+ ImmutableSegment immutableSegment = immutableSegmentDataManager.getSegment();
+ ImmutableSegmentImpl immutableSegmentImpl = (ImmutableSegmentImpl) immutableSegment;
+ Map<String, ColumnIndexContainer> columnIndexContainerMap = immutableSegmentImpl.getIndexContainerMap();
+ columnIndexMap = getImmutableSegmentColumnIndexes(columnIndexContainerMap);
Review comment:
Need to keep the check before casting here
```suggestion
IndexSegment segment = segmentDataManager.getSegment();
if (segment instanceof ImmutableSegmentImpl) {
return getImmutableSegmentColumnIndexes(((ImmutableSegmentImpl) segment).getIndexContainerMap());
} else {
return null;
}
```
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -62,24 +76,63 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
columnSet = new HashSet<>(columns);
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
- segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(COLUMN_INDEX_KEY, JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
Review comment:
(nit) Suggest putting the star-tree related methods after the indexes related methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r637541268
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -128,8 +137,49 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
indexStatus.put(RANGE_INDEX, INDEX_AVAILABLE);
}
+ if (Objects.isNull(columnIndexContainer.getJsonIndex())){
+ indexStatus.put(JSON_INDEX, INDEX_NOT_AVAILABLE);
+ } else {
+ indexStatus.put(JSON_INDEX, INDEX_AVAILABLE);
+ }
+
columnIndexMap.put(entry.getKey(), indexStatus);
}
return columnIndexMap;
}
+
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<StarTreeV2> starTrees = segmentDataManager.getSegment().getStarTrees();
+ return starTrees != null ? getImmutableSegmentStartreeIndexes(starTrees) : null;
+ }
+
+ /**
+ * Helper to loop over star trees of a segment to create a map containing star tree details.
+ */
+ private static List<Map<String, Object>> getImmutableSegmentStartreeIndexes(List<StarTreeV2> starTrees){
Review comment:
This is done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#issuecomment-840887487
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6873](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a2ce4a4) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/b8a92c41e3d063c436e37965ade577ff8e2b5f86?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b8a92c4) will **decrease** coverage by `0.09%`.
> The diff coverage is `21.42%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6873 +/- ##
============================================
- Coverage 73.42% 73.32% -0.10%
Complexity 12 12
============================================
Files 1441 1441
Lines 71431 71449 +18
Branches 10351 10354 +3
============================================
- Hits 52446 52390 -56
- Misses 15464 15544 +80
+ Partials 3521 3515 -6
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `42.03% <0.00%> (-0.09%)` | :arrow_down: |
| unittests | `65.54% <21.42%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...t/server/api/resources/SegmentMetadataFetcher.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvYXBpL3Jlc291cmNlcy9TZWdtZW50TWV0YWRhdGFGZXRjaGVyLmphdmE=) | `45.00% <21.42%> (-24.05%)` | :arrow_down: |
| [...data/manager/realtime/SegmentCommitterFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VnbWVudENvbW1pdHRlckZhY3RvcnkuamF2YQ==) | `64.70% <0.00%> (-35.30%)` | :arrow_down: |
| [...nction/DistinctCountBitmapAggregationFunction.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9EaXN0aW5jdENvdW50Qml0bWFwQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `41.75% <0.00%> (-13.92%)` | :arrow_down: |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `48.54% <0.00%> (-8.74%)` | :arrow_down: |
| [...er/api/resources/LLCSegmentCompletionHandlers.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL0xMQ1NlZ21lbnRDb21wbGV0aW9uSGFuZGxlcnMuamF2YQ==) | `51.16% <0.00%> (-8.38%)` | :arrow_down: |
| [...e/data/manager/realtime/SplitSegmentCommitter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU3BsaXRTZWdtZW50Q29tbWl0dGVyLmphdmE=) | `53.84% <0.00%> (-7.70%)` | :arrow_down: |
| [...mpl/dictionary/DoubleOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT2ZmSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=) | `59.57% <0.00%> (-4.26%)` | :arrow_down: |
| [...ller/validation/OfflineSegmentIntervalChecker.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL09mZmxpbmVTZWdtZW50SW50ZXJ2YWxDaGVja2VyLmphdmE=) | `86.25% <0.00%> (-3.75%)` | :arrow_down: |
| [.../java/org/apache/pinot/spi/data/TimeFieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9UaW1lRmllbGRTcGVjLmphdmE=) | `88.63% <0.00%> (-2.28%)` | :arrow_down: |
| [...r/validation/RealtimeSegmentValidationManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1JlYWx0aW1lU2VnbWVudFZhbGlkYXRpb25NYW5hZ2VyLmphdmE=) | `82.25% <0.00%> (-1.62%)` | :arrow_down: |
| ... and [18 more](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b8a92c4...a2ce4a4](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r633055319
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -62,24 +76,63 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
columnSet = new HashSet<>(columns);
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
- segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(COLUMN_INDEX_KEY, JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+ ImmutableSegmentDataManager immutableSegmentDataManager = (ImmutableSegmentDataManager) segmentDataManager;
+ ImmutableSegment immutableSegment = immutableSegmentDataManager.getSegment();
+ List<StarTreeV2> starTrees = immutableSegment.getStarTrees();
+ if (starTrees == null) {
+ return startreeDetails;
+ }
+ startreeDetails = getImmutableSegmentStartreeIndexes(starTrees);
+ return startreeDetails;
Review comment:
This is done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r625320057
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -63,9 +77,67 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+
+ if (segmentDataManager instanceof ImmutableSegmentDataManager) {
+ ImmutableSegmentDataManager immutableSegmentDataManager = (ImmutableSegmentDataManager) segmentDataManager;
+
+ ImmutableSegment immutableSegment = immutableSegmentDataManager.getSegment();
+
+ if (immutableSegment instanceof ImmutableSegmentImpl) {
+ List<StarTreeV2> starTrees = immutableSegment.getStarTrees();
+
+ if (starTrees == null) {
+ return startreeDetails;
+ }
+
+ startreeDetails = getImmutableSegmentStartreeIndexes(starTrees);
+ }
+ }
+ return startreeDetails;
+ }
+
+ /**
+ * Helper to loop over star trees of a segment to create a map containing star tree details.
+ */
+ private static List<Map<String, Object>> getImmutableSegmentStartreeIndexes(List<StarTreeV2> starTrees){
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+ for (StarTreeV2 starTree : starTrees) {
+ StarTreeV2Metadata starTreeMetadata = starTree.getMetadata();
+
+ Map<String, Object> starTreeIndexMap = new LinkedHashMap<>();
+
+ List<String> starTreeDimensions = starTreeMetadata.getDimensionsSplitOrder();
+ starTreeIndexMap.put(STAR_TREE_DIMENSION_COLUMNS, starTreeDimensions);
+
+ List<Map<String, String>> starTreeMetricAggregations = new ArrayList<>();
+ Set<AggregationFunctionColumnPair> functionColumnPairs = starTreeMetadata.getFunctionColumnPairs();
+ for (AggregationFunctionColumnPair functionColumnPair : functionColumnPairs) {
Review comment:
Directly put it as a list of `functionColumnPair.toColumnName()`, no need to create a separate map
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -63,9 +77,67 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+
+ if (segmentDataManager instanceof ImmutableSegmentDataManager) {
Review comment:
We don't need these `instanceof` checks. The method is declared in the interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r640132726
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -128,8 +139,50 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
indexStatus.put(RANGE_INDEX, INDEX_AVAILABLE);
}
+ if (Objects.isNull(columnIndexContainer.getJsonIndex())){
+ indexStatus.put(JSON_INDEX, INDEX_NOT_AVAILABLE);
+ } else {
+ indexStatus.put(JSON_INDEX, INDEX_AVAILABLE);
+ }
+
columnIndexMap.put(entry.getKey(), indexStatus);
}
return columnIndexMap;
}
+
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ @Nullable
+ private static List<Map<String, Object>> getStarTreeIndexesForSegment(SegmentDataManager segmentDataManager) {
+ List<StarTreeV2> starTrees = segmentDataManager.getSegment().getStarTrees();
+ return starTrees != null ? getStartreeIndexes(starTrees) : null;
+ }
+
+ /**
+ * Helper to loop over star trees of a segment to create a map containing star tree details.
+ */
+ private static List<Map<String, Object>> getStartreeIndexes(List<StarTreeV2> starTrees){
Review comment:
Fixed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r640118402
##########
File path: pinot-server/src/test/java/org/apache/pinot/server/api/TablesResourceTest.java
##########
@@ -273,6 +273,7 @@ public void testOfflineTableSegmentMetadata()
.get(String.class));
Assert.assertEquals(jsonResponse.get("columns").size(), 2);
Assert.assertEquals(jsonResponse.get("indexes").size(), 17);
+ Assert.assertEquals(jsonResponse.get("star-tree-index").size(), 0);
Review comment:
Reverted this. This test case is passing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r632098775
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -63,9 +77,67 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+
+ if (segmentDataManager instanceof ImmutableSegmentDataManager) {
Review comment:
This is done.
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -63,9 +77,67 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+
+ if (segmentDataManager instanceof ImmutableSegmentDataManager) {
+ ImmutableSegmentDataManager immutableSegmentDataManager = (ImmutableSegmentDataManager) segmentDataManager;
+
+ ImmutableSegment immutableSegment = immutableSegmentDataManager.getSegment();
+
+ if (immutableSegment instanceof ImmutableSegmentImpl) {
+ List<StarTreeV2> starTrees = immutableSegment.getStarTrees();
+
+ if (starTrees == null) {
+ return startreeDetails;
+ }
+
+ startreeDetails = getImmutableSegmentStartreeIndexes(starTrees);
+ }
+ }
+ return startreeDetails;
+ }
+
+ /**
+ * Helper to loop over star trees of a segment to create a map containing star tree details.
+ */
+ private static List<Map<String, Object>> getImmutableSegmentStartreeIndexes(List<StarTreeV2> starTrees){
+ List<Map<String, Object>> startreeDetails = new ArrayList<>();
+ for (StarTreeV2 starTree : starTrees) {
+ StarTreeV2Metadata starTreeMetadata = starTree.getMetadata();
+
+ Map<String, Object> starTreeIndexMap = new LinkedHashMap<>();
+
+ List<String> starTreeDimensions = starTreeMetadata.getDimensionsSplitOrder();
+ starTreeIndexMap.put(STAR_TREE_DIMENSION_COLUMNS, starTreeDimensions);
+
+ List<Map<String, String>> starTreeMetricAggregations = new ArrayList<>();
+ Set<AggregationFunctionColumnPair> functionColumnPairs = starTreeMetadata.getFunctionColumnPairs();
+ for (AggregationFunctionColumnPair functionColumnPair : functionColumnPairs) {
Review comment:
This is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
npawar commented on pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#issuecomment-847446140
Hey @kriti-sc please resolve the merge conflict and fix the test, then i'll merge this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r637581583
##########
File path: pinot-server/src/test/java/org/apache/pinot/server/api/TablesResourceTest.java
##########
@@ -273,6 +273,7 @@ public void testOfflineTableSegmentMetadata()
.get(String.class));
Assert.assertEquals(jsonResponse.get("columns").size(), 2);
Assert.assertEquals(jsonResponse.get("indexes").size(), 17);
+ Assert.assertEquals(jsonResponse.get("star-tree-index").size(), 0);
Review comment:
Fixed this by using `Assert.assertNull`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r640118402
##########
File path: pinot-server/src/test/java/org/apache/pinot/server/api/TablesResourceTest.java
##########
@@ -273,6 +273,7 @@ public void testOfflineTableSegmentMetadata()
.get(String.class));
Assert.assertEquals(jsonResponse.get("columns").size(), 2);
Assert.assertEquals(jsonResponse.get("indexes").size(), 17);
+ Assert.assertEquals(jsonResponse.get("star-tree-index").size(), 0);
Review comment:
Reverted this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#issuecomment-849171984
@Jackie-Jiang the tests are passing, this PR is ready to be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r637541110
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -62,25 +75,21 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
columnSet = new HashSet<>(columns);
}
ObjectNode segmentMetadataJson = (ObjectNode) segmentMetadata.toJson(columnSet);
- segmentMetadataJson.set("indexes", JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(COLUMN_INDEX_KEY, JsonUtils.objectToJsonNode(getIndexesForSegmentColumns(segmentDataManager)));
+ segmentMetadataJson.set(STAR_TREE_INDEX_KEY, JsonUtils.objectToJsonNode((getStartreeIndexForSegmentColumns(segmentDataManager))));
return JsonUtils.objectToString(segmentMetadataJson);
}
/**
* Get the JSON object with the segment column's indexing metadata.
*/
private static Map<String, Map<String, String>> getIndexesForSegmentColumns(SegmentDataManager segmentDataManager) {
Review comment:
This is done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r639981929
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -128,8 +139,50 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
indexStatus.put(RANGE_INDEX, INDEX_AVAILABLE);
}
+ if (Objects.isNull(columnIndexContainer.getJsonIndex())){
+ indexStatus.put(JSON_INDEX, INDEX_NOT_AVAILABLE);
+ } else {
+ indexStatus.put(JSON_INDEX, INDEX_AVAILABLE);
+ }
+
columnIndexMap.put(entry.getKey(), indexStatus);
}
return columnIndexMap;
}
+
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ @Nullable
+ private static List<Map<String, Object>> getStarTreeIndexesForSegment(SegmentDataManager segmentDataManager) {
+ List<StarTreeV2> starTrees = segmentDataManager.getSegment().getStarTrees();
+ return starTrees != null ? getStartreeIndexes(starTrees) : null;
+ }
+
+ /**
+ * Helper to loop over star trees of a segment to create a map containing star tree details.
+ */
+ private static List<Map<String, Object>> getStartreeIndexes(List<StarTreeV2> starTrees){
Review comment:
(nit)
```suggestion
private static List<Map<String, Object>> getStarTreeIndexes(List<StarTreeV2> starTrees){
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r637572697
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -128,8 +137,49 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
indexStatus.put(RANGE_INDEX, INDEX_AVAILABLE);
}
+ if (Objects.isNull(columnIndexContainer.getJsonIndex())){
+ indexStatus.put(JSON_INDEX, INDEX_NOT_AVAILABLE);
+ } else {
+ indexStatus.put(JSON_INDEX, INDEX_AVAILABLE);
+ }
+
columnIndexMap.put(entry.getKey(), indexStatus);
}
return columnIndexMap;
}
+
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<StarTreeV2> starTrees = segmentDataManager.getSegment().getStarTrees();
+ return starTrees != null ? getImmutableSegmentStartreeIndexes(starTrees) : null;
+ }
+
+ /**
+ * Helper to loop over star trees of a segment to create a map containing star tree details.
+ */
+ private static List<Map<String, Object>> getImmutableSegmentStartreeIndexes(List<StarTreeV2> starTrees){
Review comment:
This is resolved in a different way than suggested, please do take a look. @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kriti-sc commented on a change in pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
kriti-sc commented on a change in pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#discussion_r637541268
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/api/resources/SegmentMetadataFetcher.java
##########
@@ -128,8 +137,49 @@ public static String getSegmentMetadata(SegmentDataManager segmentDataManager, L
indexStatus.put(RANGE_INDEX, INDEX_AVAILABLE);
}
+ if (Objects.isNull(columnIndexContainer.getJsonIndex())){
+ indexStatus.put(JSON_INDEX, INDEX_NOT_AVAILABLE);
+ } else {
+ indexStatus.put(JSON_INDEX, INDEX_AVAILABLE);
+ }
+
columnIndexMap.put(entry.getKey(), indexStatus);
}
return columnIndexMap;
}
+
+ /**
+ * Get the JSON object containing star tree index details for a segment.
+ */
+ private static List<Map<String, Object>> getStartreeIndexForSegmentColumns(SegmentDataManager segmentDataManager) {
+ List<StarTreeV2> starTrees = segmentDataManager.getSegment().getStarTrees();
+ return starTrees != null ? getImmutableSegmentStartreeIndexes(starTrees) : null;
+ }
+
+ /**
+ * Helper to loop over star trees of a segment to create a map containing star tree details.
+ */
+ private static List<Map<String, Object>> getImmutableSegmentStartreeIndexes(List<StarTreeV2> starTrees){
Review comment:
~~This is done~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6873: feature/#6766 JSON and Startree index information in API
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6873:
URL: https://github.com/apache/incubator-pinot/pull/6873#issuecomment-840887487
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#6873](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5791012) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/d5e7106546f0b134a29028b2065bba28929d5e8e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d5e7106) will **decrease** coverage by `30.84%`.
> The diff coverage is `35.01%`.
> :exclamation: Current head 5791012 differs from pull request most recent head 2f615b5. Consider uploading reports for the commit 2f615b5 to get more accurate results
[](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #6873 +/- ##
=============================================
- Coverage 74.05% 43.21% -30.85%
+ Complexity 12 7 -5
=============================================
Files 1421 1432 +11
Lines 69141 70621 +1480
Branches 9986 10209 +223
=============================================
- Hits 51204 30520 -20684
- Misses 14590 37480 +22890
+ Partials 3347 2621 -726
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| integration | `43.21% <35.01%> (-0.43%)` | `7.00 <0.00> (ø)` | |
| unittests | `?` | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...thandler/SingleConnectionBrokerRequestHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvU2luZ2xlQ29ubmVjdGlvbkJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `85.41% <ø> (ø)` | `0.00 <0.00> (ø)` | |
| [...roker/routing/segmentpruner/TimeSegmentPruner.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL1RpbWVTZWdtZW50UHJ1bmVyLmphdmE=) | `0.00% <0.00%> (-88.24%)` | `0.00 <0.00> (ø)` | |
| [...roker/routing/segmentpruner/interval/Interval.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL2ludGVydmFsL0ludGVydmFsLmphdmE=) | `0.00% <0.00%> (-79.49%)` | `0.00 <0.00> (ø)` | |
| [...uting/segmentselector/RealtimeSegmentSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50c2VsZWN0b3IvUmVhbHRpbWVTZWdtZW50U2VsZWN0b3IuamF2YQ==) | `82.81% <0.00%> (-13.97%)` | `0.00 <0.00> (ø)` | |
| [...inot/common/function/scalar/DateTimeFunctions.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0RhdGVUaW1lRnVuY3Rpb25zLmphdmE=) | `4.22% <0.00%> (-92.88%)` | `0.00 <0.00> (ø)` | |
| [...a/org/apache/pinot/common/metrics/ServerGauge.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9TZXJ2ZXJHYXVnZS5qYXZh) | `95.83% <ø> (-0.17%)` | `0.00 <0.00> (ø)` | |
| [...org/apache/pinot/common/utils/PinotAppConfigs.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvUGlub3RBcHBDb25maWdzLmphdmE=) | `0.00% <0.00%> (-65.26%)` | `0.00 <0.00> (ø)` | |
| [...inot/controller/api/access/AccessControlUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvYWNjZXNzL0FjY2Vzc0NvbnRyb2xVdGlscy5qYXZh) | `80.00% <ø> (ø)` | `0.00 <0.00> (ø)` | |
| [...che/pinot/controller/api/debug/TableDebugInfo.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvZGVidWcvVGFibGVEZWJ1Z0luZm8uamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [.../api/exception/ControllerApplicationException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvZXhjZXB0aW9uL0NvbnRyb2xsZXJBcHBsaWNhdGlvbkV4Y2VwdGlvbi5qYXZh) | `60.00% <ø> (ø)` | `0.00 <0.00> (?)` | |
| ... and [1079 more](https://codecov.io/gh/apache/incubator-pinot/pull/6873/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d5e7106...2f615b5](https://codecov.io/gh/apache/incubator-pinot/pull/6873?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org