You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/08/04 00:51:43 UTC
[GitHub] [pinot] Jackie-Jiang opened a new pull request, #9159: Add id to MetadataKey so that we can change the key order
Jackie-Jiang opened a new pull request, #9159:
URL: https://github.com/apache/pinot/pull/9159
Assign an id to each `MetadataKey` in the `DataTable` so that we can change the key order (insert new keys) if necessary for readability. There have been multiple issues caused by people adding keys in the middle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a diff in pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
richardstartin commented on code in PR #9159:
URL: https://github.com/apache/pinot/pull/9159#discussion_r938129373
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java:
##########
@@ -90,64 +90,72 @@ enum MetadataValueType {
INT, LONG, STRING
}
- /* The MetadataKey is used in V3, where we present metadata as Map<MetadataKey, String>
+ /* The MetadataKey is used since V3, where we present metadata as Map<MetadataKey, String>
* ATTENTION:
- * - Don't change existing keys.
- * - Don't remove existing keys.
- * - Always add new keys to the end.
+ * - Don't change the id of existing keys.
+ * - Always add new keys with higher ids.
* Otherwise, backward compatibility will be broken.
*/
enum MetadataKey {
- UNKNOWN("unknown", MetadataValueType.STRING),
- TABLE("table", MetadataValueType.STRING), // NOTE: this key is only used in PrioritySchedulerTest
- NUM_DOCS_SCANNED("numDocsScanned", MetadataValueType.LONG),
- NUM_ENTRIES_SCANNED_IN_FILTER("numEntriesScannedInFilter", MetadataValueType.LONG),
- NUM_ENTRIES_SCANNED_POST_FILTER("numEntriesScannedPostFilter", MetadataValueType.LONG),
- NUM_SEGMENTS_QUERIED("numSegmentsQueried", MetadataValueType.INT),
- NUM_SEGMENTS_PROCESSED("numSegmentsProcessed", MetadataValueType.INT),
- NUM_SEGMENTS_MATCHED("numSegmentsMatched", MetadataValueType.INT),
- NUM_CONSUMING_SEGMENTS_QUERIED("numConsumingSegmentsQueried", MetadataValueType.INT),
- MIN_CONSUMING_FRESHNESS_TIME_MS("minConsumingFreshnessTimeMs", MetadataValueType.LONG),
- TOTAL_DOCS("totalDocs", MetadataValueType.LONG),
- NUM_GROUPS_LIMIT_REACHED("numGroupsLimitReached", MetadataValueType.STRING),
- TIME_USED_MS("timeUsedMs", MetadataValueType.LONG),
- TRACE_INFO("traceInfo", MetadataValueType.STRING),
- REQUEST_ID("requestId", MetadataValueType.LONG),
- NUM_RESIZES("numResizes", MetadataValueType.INT),
- RESIZE_TIME_MS("resizeTimeMs", MetadataValueType.LONG),
- THREAD_CPU_TIME_NS("threadCpuTimeNs", MetadataValueType.LONG),
- SYSTEM_ACTIVITIES_CPU_TIME_NS("systemActivitiesCpuTimeNs", MetadataValueType.LONG),
- RESPONSE_SER_CPU_TIME_NS("responseSerializationCpuTimeNs", MetadataValueType.LONG),
- NUM_SEGMENTS_PRUNED_BY_SERVER("numSegmentsPrunedByServer", MetadataValueType.INT),
- NUM_SEGMENTS_PRUNED_INVALID("numSegmentsPrunedByInvalid", MetadataValueType.INT),
- NUM_SEGMENTS_PRUNED_BY_LIMIT("numSegmentsPrunedByLimit", MetadataValueType.INT),
- NUM_SEGMENTS_PRUNED_BY_VALUE("numSegmentsPrunedByValue", MetadataValueType.INT),
- EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS("explainPlanNumEmptyFilterSegments", MetadataValueType.INT),
- EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS("explainPlanNumMatchAllFilterSegments", MetadataValueType.INT),
- NUM_CONSUMING_SEGMENTS_PROCESSED("numConsumingSegmentsProcessed", MetadataValueType.INT),
- NUM_CONSUMING_SEGMENTS_MATCHED("numConsumingSegmentsMatched", MetadataValueType.INT);
-
- private static final MetadataKey[] VALUES;
+ UNKNOWN(0, "unknown", MetadataValueType.STRING),
+ TABLE(1, "table", MetadataValueType.STRING), // NOTE: this key is only used in PrioritySchedulerTest
+ NUM_DOCS_SCANNED(2, "numDocsScanned", MetadataValueType.LONG),
+ NUM_ENTRIES_SCANNED_IN_FILTER(3, "numEntriesScannedInFilter", MetadataValueType.LONG),
+ NUM_ENTRIES_SCANNED_POST_FILTER(4, "numEntriesScannedPostFilter", MetadataValueType.LONG),
+ NUM_SEGMENTS_QUERIED(5, "numSegmentsQueried", MetadataValueType.INT),
+ NUM_SEGMENTS_PROCESSED(6, "numSegmentsProcessed", MetadataValueType.INT),
+ NUM_SEGMENTS_MATCHED(7, "numSegmentsMatched", MetadataValueType.INT),
+ NUM_CONSUMING_SEGMENTS_QUERIED(8, "numConsumingSegmentsQueried", MetadataValueType.INT),
+ NUM_CONSUMING_SEGMENTS_PROCESSED(26, "numConsumingSegmentsProcessed", MetadataValueType.INT),
+ NUM_CONSUMING_SEGMENTS_MATCHED(27, "numConsumingSegmentsMatched", MetadataValueType.INT),
+ MIN_CONSUMING_FRESHNESS_TIME_MS(9, "minConsumingFreshnessTimeMs", MetadataValueType.LONG),
+ TOTAL_DOCS(10, "totalDocs", MetadataValueType.LONG),
+ NUM_GROUPS_LIMIT_REACHED(11, "numGroupsLimitReached", MetadataValueType.STRING),
+ TIME_USED_MS(12, "timeUsedMs", MetadataValueType.LONG),
+ TRACE_INFO(13, "traceInfo", MetadataValueType.STRING),
+ REQUEST_ID(14, "requestId", MetadataValueType.LONG),
+ NUM_RESIZES(15, "numResizes", MetadataValueType.INT),
+ RESIZE_TIME_MS(16, "resizeTimeMs", MetadataValueType.LONG),
+ THREAD_CPU_TIME_NS(17, "threadCpuTimeNs", MetadataValueType.LONG),
+ SYSTEM_ACTIVITIES_CPU_TIME_NS(18, "systemActivitiesCpuTimeNs", MetadataValueType.LONG),
+ RESPONSE_SER_CPU_TIME_NS(19, "responseSerializationCpuTimeNs", MetadataValueType.LONG),
+ NUM_SEGMENTS_PRUNED_BY_SERVER(20, "numSegmentsPrunedByServer", MetadataValueType.INT),
+ NUM_SEGMENTS_PRUNED_INVALID(21, "numSegmentsPrunedByInvalid", MetadataValueType.INT),
+ NUM_SEGMENTS_PRUNED_BY_LIMIT(22, "numSegmentsPrunedByLimit", MetadataValueType.INT),
+ NUM_SEGMENTS_PRUNED_BY_VALUE(23, "numSegmentsPrunedByValue", MetadataValueType.INT),
+ EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS(24, "explainPlanNumEmptyFilterSegments", MetadataValueType.INT),
+ EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS(25, "explainPlanNumMatchAllFilterSegments", MetadataValueType.INT);
+
+ private static final MetadataKey[] ID_TO_ENUM_KEY_MAP;
private static final Map<String, MetadataKey> NAME_TO_ENUM_KEY_MAP = new HashMap<>();
+
+ private final int _id;
private final String _name;
private final MetadataValueType _valueType;
- MetadataKey(String name, MetadataValueType valueType) {
+ MetadataKey(int id, String name, MetadataValueType valueType) {
+ _id = id;
_name = name;
_valueType = valueType;
}
- // getByOrdinal returns an enum key for a given ordinal or null if the key does not exist.
+ /**
+ * Returns the MetadataKey for the given id, or {@code null} if the id does not exist.
+ */
@Nullable
- public static MetadataKey getByOrdinal(int ordinal) {
- return VALUES[Math.min(ordinal, VALUES.length - 1)];
+ public static MetadataKey getById(int id) {
+ return id < ID_TO_ENUM_KEY_MAP.length ? ID_TO_ENUM_KEY_MAP[id] : null;
Review Comment:
Ok, it’s better now it supports negative values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on PR #9159:
URL: https://github.com/apache/pinot/pull/9159#issuecomment-1205620358
@siddharthteotia @richardstartin
To add some context here, we had got multiple PRs merged without following the convention, which broke the backward compatibility. It is hard for all the contributors and reviewers to remember the convention, especially when the change is not next to the comment describing how to keep backward compatible. As @FelixGV pointed out, using enum's ordinal is error-prone, and we need to use a way to enforce people to follow the convention, and make it easy for reviewers to catch when the convention is not followed.
Example PRs that breaks the backward compatibility:
#9092 Merged then reverted the next day
#8884 Merged but not caught the backward compatibility issue until this PR (already one month!). I actually pointed it out during the [review](https://github.com/apache/pinot/pull/8884#discussion_r905529608), but the fix goes to the wrong class and I didn't get a chance to review it again. This is a proof that we don't have any guard for this convention, except for the comment that is 30 lines away from the actual change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] FelixGV commented on pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
FelixGV commented on PR #9159:
URL: https://github.com/apache/pinot/pull/9159#issuecomment-1205211259
> > Part of me thinks that people really need to read the code (javadoc just above the enum) before making bad changes of inserting keys in the middle. So, we should try to stick to current mechanism if possible.
>
> +1. Enums should be treated as append-only code constructs.
Following conventions is a good convention, but making the code structure enforce or incentivize conventions may be even better.
IMHO, it's not too controversial to add a manually maintained ID for enums that need to be sent across processes or persisted across restarts of the same process (i.e. any enum that will potentially be produced by one version of the software and consumed by another).
Using the enum's built-in ordinal is error-prone, since it doesn’t survive re-ordering, nor insertions and removals from the middle. Using a manually-maintained ID on the other hand is resilient to all these scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on PR #9159:
URL: https://github.com/apache/pinot/pull/9159#issuecomment-1206074320
Let me go through the change once today/tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
richardstartin commented on PR #9159:
URL: https://github.com/apache/pinot/pull/9159#issuecomment-1205082213
> Part of me thinks that people really need to read the code (javadoc just above the enum) before making bad changes of inserting keys in the middle. So, we should try to stick to current mechanism if possible.
+1. Enums should be treated as append-only code constructs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged PR #9159:
URL: https://github.com/apache/pinot/pull/9159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a diff in pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
richardstartin commented on code in PR #9159:
URL: https://github.com/apache/pinot/pull/9159#discussion_r937629134
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java:
##########
@@ -90,64 +90,72 @@ enum MetadataValueType {
INT, LONG, STRING
}
- /* The MetadataKey is used in V3, where we present metadata as Map<MetadataKey, String>
+ /* The MetadataKey is used since V3, where we present metadata as Map<MetadataKey, String>
* ATTENTION:
- * - Don't change existing keys.
- * - Don't remove existing keys.
- * - Always add new keys to the end.
+ * - Don't change the id of existing keys.
+ * - Always add new keys with higher ids.
* Otherwise, backward compatibility will be broken.
*/
enum MetadataKey {
- UNKNOWN("unknown", MetadataValueType.STRING),
- TABLE("table", MetadataValueType.STRING), // NOTE: this key is only used in PrioritySchedulerTest
- NUM_DOCS_SCANNED("numDocsScanned", MetadataValueType.LONG),
- NUM_ENTRIES_SCANNED_IN_FILTER("numEntriesScannedInFilter", MetadataValueType.LONG),
- NUM_ENTRIES_SCANNED_POST_FILTER("numEntriesScannedPostFilter", MetadataValueType.LONG),
- NUM_SEGMENTS_QUERIED("numSegmentsQueried", MetadataValueType.INT),
- NUM_SEGMENTS_PROCESSED("numSegmentsProcessed", MetadataValueType.INT),
- NUM_SEGMENTS_MATCHED("numSegmentsMatched", MetadataValueType.INT),
- NUM_CONSUMING_SEGMENTS_QUERIED("numConsumingSegmentsQueried", MetadataValueType.INT),
- MIN_CONSUMING_FRESHNESS_TIME_MS("minConsumingFreshnessTimeMs", MetadataValueType.LONG),
- TOTAL_DOCS("totalDocs", MetadataValueType.LONG),
- NUM_GROUPS_LIMIT_REACHED("numGroupsLimitReached", MetadataValueType.STRING),
- TIME_USED_MS("timeUsedMs", MetadataValueType.LONG),
- TRACE_INFO("traceInfo", MetadataValueType.STRING),
- REQUEST_ID("requestId", MetadataValueType.LONG),
- NUM_RESIZES("numResizes", MetadataValueType.INT),
- RESIZE_TIME_MS("resizeTimeMs", MetadataValueType.LONG),
- THREAD_CPU_TIME_NS("threadCpuTimeNs", MetadataValueType.LONG),
- SYSTEM_ACTIVITIES_CPU_TIME_NS("systemActivitiesCpuTimeNs", MetadataValueType.LONG),
- RESPONSE_SER_CPU_TIME_NS("responseSerializationCpuTimeNs", MetadataValueType.LONG),
- NUM_SEGMENTS_PRUNED_BY_SERVER("numSegmentsPrunedByServer", MetadataValueType.INT),
- NUM_SEGMENTS_PRUNED_INVALID("numSegmentsPrunedByInvalid", MetadataValueType.INT),
- NUM_SEGMENTS_PRUNED_BY_LIMIT("numSegmentsPrunedByLimit", MetadataValueType.INT),
- NUM_SEGMENTS_PRUNED_BY_VALUE("numSegmentsPrunedByValue", MetadataValueType.INT),
- EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS("explainPlanNumEmptyFilterSegments", MetadataValueType.INT),
- EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS("explainPlanNumMatchAllFilterSegments", MetadataValueType.INT),
- NUM_CONSUMING_SEGMENTS_PROCESSED("numConsumingSegmentsProcessed", MetadataValueType.INT),
- NUM_CONSUMING_SEGMENTS_MATCHED("numConsumingSegmentsMatched", MetadataValueType.INT);
-
- private static final MetadataKey[] VALUES;
+ UNKNOWN(0, "unknown", MetadataValueType.STRING),
+ TABLE(1, "table", MetadataValueType.STRING), // NOTE: this key is only used in PrioritySchedulerTest
+ NUM_DOCS_SCANNED(2, "numDocsScanned", MetadataValueType.LONG),
+ NUM_ENTRIES_SCANNED_IN_FILTER(3, "numEntriesScannedInFilter", MetadataValueType.LONG),
+ NUM_ENTRIES_SCANNED_POST_FILTER(4, "numEntriesScannedPostFilter", MetadataValueType.LONG),
+ NUM_SEGMENTS_QUERIED(5, "numSegmentsQueried", MetadataValueType.INT),
+ NUM_SEGMENTS_PROCESSED(6, "numSegmentsProcessed", MetadataValueType.INT),
+ NUM_SEGMENTS_MATCHED(7, "numSegmentsMatched", MetadataValueType.INT),
+ NUM_CONSUMING_SEGMENTS_QUERIED(8, "numConsumingSegmentsQueried", MetadataValueType.INT),
+ NUM_CONSUMING_SEGMENTS_PROCESSED(26, "numConsumingSegmentsProcessed", MetadataValueType.INT),
+ NUM_CONSUMING_SEGMENTS_MATCHED(27, "numConsumingSegmentsMatched", MetadataValueType.INT),
+ MIN_CONSUMING_FRESHNESS_TIME_MS(9, "minConsumingFreshnessTimeMs", MetadataValueType.LONG),
+ TOTAL_DOCS(10, "totalDocs", MetadataValueType.LONG),
+ NUM_GROUPS_LIMIT_REACHED(11, "numGroupsLimitReached", MetadataValueType.STRING),
+ TIME_USED_MS(12, "timeUsedMs", MetadataValueType.LONG),
+ TRACE_INFO(13, "traceInfo", MetadataValueType.STRING),
+ REQUEST_ID(14, "requestId", MetadataValueType.LONG),
+ NUM_RESIZES(15, "numResizes", MetadataValueType.INT),
+ RESIZE_TIME_MS(16, "resizeTimeMs", MetadataValueType.LONG),
+ THREAD_CPU_TIME_NS(17, "threadCpuTimeNs", MetadataValueType.LONG),
+ SYSTEM_ACTIVITIES_CPU_TIME_NS(18, "systemActivitiesCpuTimeNs", MetadataValueType.LONG),
+ RESPONSE_SER_CPU_TIME_NS(19, "responseSerializationCpuTimeNs", MetadataValueType.LONG),
+ NUM_SEGMENTS_PRUNED_BY_SERVER(20, "numSegmentsPrunedByServer", MetadataValueType.INT),
+ NUM_SEGMENTS_PRUNED_INVALID(21, "numSegmentsPrunedByInvalid", MetadataValueType.INT),
+ NUM_SEGMENTS_PRUNED_BY_LIMIT(22, "numSegmentsPrunedByLimit", MetadataValueType.INT),
+ NUM_SEGMENTS_PRUNED_BY_VALUE(23, "numSegmentsPrunedByValue", MetadataValueType.INT),
+ EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS(24, "explainPlanNumEmptyFilterSegments", MetadataValueType.INT),
+ EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS(25, "explainPlanNumMatchAllFilterSegments", MetadataValueType.INT);
+
+ private static final MetadataKey[] ID_TO_ENUM_KEY_MAP;
private static final Map<String, MetadataKey> NAME_TO_ENUM_KEY_MAP = new HashMap<>();
+
+ private final int _id;
private final String _name;
private final MetadataValueType _valueType;
- MetadataKey(String name, MetadataValueType valueType) {
+ MetadataKey(int id, String name, MetadataValueType valueType) {
+ _id = id;
_name = name;
_valueType = valueType;
}
- // getByOrdinal returns an enum key for a given ordinal or null if the key does not exist.
+ /**
+ * Returns the MetadataKey for the given id, or {@code null} if the id does not exist.
+ */
@Nullable
- public static MetadataKey getByOrdinal(int ordinal) {
- return VALUES[Math.min(ordinal, VALUES.length - 1)];
+ public static MetadataKey getById(int id) {
+ return id < ID_TO_ENUM_KEY_MAP.length ? ID_TO_ENUM_KEY_MAP[id] : null;
Review Comment:
This appears to be an unnecessary change as there is a null element at the end of the map to deal with unmapped ordinals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
richardstartin commented on PR #9159:
URL: https://github.com/apache/pinot/pull/9159#issuecomment-1205626564
> @siddharthteotia @richardstartin To add some context here, we had got multiple PRs merged without following the convention, which broke the backward compatibility. It is hard for all the contributors and reviewers to remember the convention, especially when the change is not next to the comment describing how to keep backward compatible. As @FelixGV pointed out, using enum's ordinal is error-prone, and we need to use a way to enforce people to follow the convention, and make it easy for reviewers to catch when the convention is not followed.
>
> Example PRs that breaks the backward compatibility: #9092 Merged then reverted the next day #8884 Merged but not caught the backward compatibility issue until this PR (already one month!). I actually pointed it out during the [review](https://github.com/apache/pinot/pull/8884#discussion_r905529608), but the fix goes to the wrong class and I didn't get a chance to review it again. This is a proof that we don't have any guard for this convention, except for the comment that is 30 lines away from the actual change.
Clearly a very simple regression test can prevent changes like this from getting merged (and from someone accidentally adding a duplicate id, against which there are currently no guards). Code review shouldn’t focus on things which can be automated simply because hardly anyone has (or should have) the attention span to verify trivialities like these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9159:
URL: https://github.com/apache/pinot/pull/9159#discussion_r938105647
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java:
##########
@@ -90,64 +90,72 @@ enum MetadataValueType {
INT, LONG, STRING
}
- /* The MetadataKey is used in V3, where we present metadata as Map<MetadataKey, String>
+ /* The MetadataKey is used since V3, where we present metadata as Map<MetadataKey, String>
* ATTENTION:
- * - Don't change existing keys.
- * - Don't remove existing keys.
- * - Always add new keys to the end.
+ * - Don't change the id of existing keys.
+ * - Always add new keys with higher ids.
* Otherwise, backward compatibility will be broken.
*/
enum MetadataKey {
- UNKNOWN("unknown", MetadataValueType.STRING),
- TABLE("table", MetadataValueType.STRING), // NOTE: this key is only used in PrioritySchedulerTest
- NUM_DOCS_SCANNED("numDocsScanned", MetadataValueType.LONG),
- NUM_ENTRIES_SCANNED_IN_FILTER("numEntriesScannedInFilter", MetadataValueType.LONG),
- NUM_ENTRIES_SCANNED_POST_FILTER("numEntriesScannedPostFilter", MetadataValueType.LONG),
- NUM_SEGMENTS_QUERIED("numSegmentsQueried", MetadataValueType.INT),
- NUM_SEGMENTS_PROCESSED("numSegmentsProcessed", MetadataValueType.INT),
- NUM_SEGMENTS_MATCHED("numSegmentsMatched", MetadataValueType.INT),
- NUM_CONSUMING_SEGMENTS_QUERIED("numConsumingSegmentsQueried", MetadataValueType.INT),
- MIN_CONSUMING_FRESHNESS_TIME_MS("minConsumingFreshnessTimeMs", MetadataValueType.LONG),
- TOTAL_DOCS("totalDocs", MetadataValueType.LONG),
- NUM_GROUPS_LIMIT_REACHED("numGroupsLimitReached", MetadataValueType.STRING),
- TIME_USED_MS("timeUsedMs", MetadataValueType.LONG),
- TRACE_INFO("traceInfo", MetadataValueType.STRING),
- REQUEST_ID("requestId", MetadataValueType.LONG),
- NUM_RESIZES("numResizes", MetadataValueType.INT),
- RESIZE_TIME_MS("resizeTimeMs", MetadataValueType.LONG),
- THREAD_CPU_TIME_NS("threadCpuTimeNs", MetadataValueType.LONG),
- SYSTEM_ACTIVITIES_CPU_TIME_NS("systemActivitiesCpuTimeNs", MetadataValueType.LONG),
- RESPONSE_SER_CPU_TIME_NS("responseSerializationCpuTimeNs", MetadataValueType.LONG),
- NUM_SEGMENTS_PRUNED_BY_SERVER("numSegmentsPrunedByServer", MetadataValueType.INT),
- NUM_SEGMENTS_PRUNED_INVALID("numSegmentsPrunedByInvalid", MetadataValueType.INT),
- NUM_SEGMENTS_PRUNED_BY_LIMIT("numSegmentsPrunedByLimit", MetadataValueType.INT),
- NUM_SEGMENTS_PRUNED_BY_VALUE("numSegmentsPrunedByValue", MetadataValueType.INT),
- EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS("explainPlanNumEmptyFilterSegments", MetadataValueType.INT),
- EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS("explainPlanNumMatchAllFilterSegments", MetadataValueType.INT),
- NUM_CONSUMING_SEGMENTS_PROCESSED("numConsumingSegmentsProcessed", MetadataValueType.INT),
- NUM_CONSUMING_SEGMENTS_MATCHED("numConsumingSegmentsMatched", MetadataValueType.INT);
-
- private static final MetadataKey[] VALUES;
+ UNKNOWN(0, "unknown", MetadataValueType.STRING),
+ TABLE(1, "table", MetadataValueType.STRING), // NOTE: this key is only used in PrioritySchedulerTest
+ NUM_DOCS_SCANNED(2, "numDocsScanned", MetadataValueType.LONG),
+ NUM_ENTRIES_SCANNED_IN_FILTER(3, "numEntriesScannedInFilter", MetadataValueType.LONG),
+ NUM_ENTRIES_SCANNED_POST_FILTER(4, "numEntriesScannedPostFilter", MetadataValueType.LONG),
+ NUM_SEGMENTS_QUERIED(5, "numSegmentsQueried", MetadataValueType.INT),
+ NUM_SEGMENTS_PROCESSED(6, "numSegmentsProcessed", MetadataValueType.INT),
+ NUM_SEGMENTS_MATCHED(7, "numSegmentsMatched", MetadataValueType.INT),
+ NUM_CONSUMING_SEGMENTS_QUERIED(8, "numConsumingSegmentsQueried", MetadataValueType.INT),
+ NUM_CONSUMING_SEGMENTS_PROCESSED(26, "numConsumingSegmentsProcessed", MetadataValueType.INT),
+ NUM_CONSUMING_SEGMENTS_MATCHED(27, "numConsumingSegmentsMatched", MetadataValueType.INT),
+ MIN_CONSUMING_FRESHNESS_TIME_MS(9, "minConsumingFreshnessTimeMs", MetadataValueType.LONG),
+ TOTAL_DOCS(10, "totalDocs", MetadataValueType.LONG),
+ NUM_GROUPS_LIMIT_REACHED(11, "numGroupsLimitReached", MetadataValueType.STRING),
+ TIME_USED_MS(12, "timeUsedMs", MetadataValueType.LONG),
+ TRACE_INFO(13, "traceInfo", MetadataValueType.STRING),
+ REQUEST_ID(14, "requestId", MetadataValueType.LONG),
+ NUM_RESIZES(15, "numResizes", MetadataValueType.INT),
+ RESIZE_TIME_MS(16, "resizeTimeMs", MetadataValueType.LONG),
+ THREAD_CPU_TIME_NS(17, "threadCpuTimeNs", MetadataValueType.LONG),
+ SYSTEM_ACTIVITIES_CPU_TIME_NS(18, "systemActivitiesCpuTimeNs", MetadataValueType.LONG),
+ RESPONSE_SER_CPU_TIME_NS(19, "responseSerializationCpuTimeNs", MetadataValueType.LONG),
+ NUM_SEGMENTS_PRUNED_BY_SERVER(20, "numSegmentsPrunedByServer", MetadataValueType.INT),
+ NUM_SEGMENTS_PRUNED_INVALID(21, "numSegmentsPrunedByInvalid", MetadataValueType.INT),
+ NUM_SEGMENTS_PRUNED_BY_LIMIT(22, "numSegmentsPrunedByLimit", MetadataValueType.INT),
+ NUM_SEGMENTS_PRUNED_BY_VALUE(23, "numSegmentsPrunedByValue", MetadataValueType.INT),
+ EXPLAIN_PLAN_NUM_EMPTY_FILTER_SEGMENTS(24, "explainPlanNumEmptyFilterSegments", MetadataValueType.INT),
+ EXPLAIN_PLAN_NUM_MATCH_ALL_FILTER_SEGMENTS(25, "explainPlanNumMatchAllFilterSegments", MetadataValueType.INT);
+
+ private static final MetadataKey[] ID_TO_ENUM_KEY_MAP;
private static final Map<String, MetadataKey> NAME_TO_ENUM_KEY_MAP = new HashMap<>();
+
+ private final int _id;
private final String _name;
private final MetadataValueType _valueType;
- MetadataKey(String name, MetadataValueType valueType) {
+ MetadataKey(int id, String name, MetadataValueType valueType) {
+ _id = id;
_name = name;
_valueType = valueType;
}
- // getByOrdinal returns an enum key for a given ordinal or null if the key does not exist.
+ /**
+ * Returns the MetadataKey for the given id, or {@code null} if the id does not exist.
+ */
@Nullable
- public static MetadataKey getByOrdinal(int ordinal) {
- return VALUES[Math.min(ordinal, VALUES.length - 1)];
+ public static MetadataKey getById(int id) {
+ return id < ID_TO_ENUM_KEY_MAP.length ? ID_TO_ENUM_KEY_MAP[id] : null;
Review Comment:
Functionality wise, it doesn't make a difference (I updated it to handle negative input as well). Readability wise, IMO it is more readable without extra overhead. It does take me some time to understand that we append a null element to the end of the array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on PR #9159:
URL: https://github.com/apache/pinot/pull/9159#issuecomment-1205669224
> Clearly a very simple regression test can prevent changes like this from getting merged (and from someone accidentally adding a duplicate id, against which there are currently no guards). Code review shouldn’t focus on things which can be automated simply because hardly anyone has (or should have) the attention span to verify trivialities like these.
We may add a regression test to verify all the existing keys still have the same ordinal, but we also need to take extra effort maintaining the test whenever a new key is added. E.g. #8884 can still happen if the previous PR didn't add the new keys to the test
We do check duplicate id in the static block.
Assigning an id has extra benefits though:
- People can change the order of the keys for readability (and that's the main reason why people tend to add new keys in the middle so that similar metadata are grouped together, e.g. `numConsumingSegmentsQueried`, `numConsumingSegmentsProcessed`, `numConsumingSegmentsMatched`)
- We may remove the old keys that no longer apply
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on PR #9159:
URL: https://github.com/apache/pinot/pull/9159#issuecomment-1204851048
Part of me thinks that people really need to read the code (javadoc just above the enum) before making bad changes of inserting keys in the middle. So, we should try to stick to current mechanism if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #9159: Add id to MetadataKey so that we can change the key order
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9159:
URL: https://github.com/apache/pinot/pull/9159#issuecomment-1204660132
# [Codecov](https://codecov.io/gh/apache/pinot/pull/9159?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9159](https://codecov.io/gh/apache/pinot/pull/9159?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d00f746) into [master](https://codecov.io/gh/apache/pinot/commit/449f89a3d7b809daa5504147d283e763d7f44a53?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (449f89a) will **increase** coverage by `35.04%`.
> The diff coverage is `93.33%`.
```diff
@@ Coverage Diff @@
## master #9159 +/- ##
=============================================
+ Coverage 28.49% 63.54% +35.04%
- Complexity 53 4736 +4683
=============================================
Files 1836 1799 -37
Lines 98103 96316 -1787
Branches 14897 14691 -206
=============================================
+ Hits 27956 61200 +33244
+ Misses 67460 30680 -36780
- Partials 2687 4436 +1749
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `67.01% <93.33%> (?)` | |
| unittests2 | `15.27% <0.00%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9159?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../java/org/apache/pinot/common/utils/DataTable.java](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVRhYmxlLmphdmE=) | `94.44% <92.68%> (-1.21%)` | :arrow_down: |
| [...che/pinot/core/common/datablock/BaseDataBlock.java](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZGF0YWJsb2NrL0Jhc2VEYXRhQmxvY2suamF2YQ==) | `83.51% <100.00%> (+30.46%)` | :arrow_up: |
| [...e/pinot/core/common/datatable/DataTableImplV3.java](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZGF0YXRhYmxlL0RhdGFUYWJsZUltcGxWMy5qYXZh) | `91.76% <100.00%> (+1.17%)` | :arrow_up: |
| [...va/org/apache/pinot/core/routing/RoutingTable.java](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yb3V0aW5nL1JvdXRpbmdUYWJsZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/common/config/NettyConfig.java](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL05ldHR5Q29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ache/pinot/server/access/AccessControlFactory.java](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL2FjY2Vzcy9BY2Nlc3NDb250cm9sRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1515 more](https://codecov.io/gh/apache/pinot/pull/9159/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: Codecov can now indicate which changes are the most critical in Pull Requests. [Learn more](https://about.codecov.io/product/feature/runtime-insights/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org