You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/11/02 21:39:29 UTC
[GitHub] [pinot] walterddr opened a new pull request, #9712: add index key and size map to segment metadata
walterddr opened a new pull request, #9712:
URL: https://github.com/apache/pinot/pull/9712
related to #8862. we also want to report these info when hitting server segment metadata endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9712: add index key and size map to segment metadata
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9712:
URL: https://github.com/apache/pinot/pull/9712#discussion_r1012324512
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -65,13 +66,14 @@ public class ColumnMetadataImpl implements ColumnMetadata {
private final int _totalNumberOfEntries;
private final PartitionFunction _partitionFunction;
private final Set<Integer> _partitions;
+ private final Map<ColumnIndexType, Long> _indexSizeMap;
Review Comment:
adding this map here should allow us to report index info (only size for now)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on pull request #9712: add index key and size map to segment metadata
Posted by GitBox <gi...@apache.org>.
walterddr commented on PR #9712:
URL: https://github.com/apache/pinot/pull/9712#issuecomment-1303953324
CC @npawar @nizarhejazi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #9712: add index key and size map to segment metadata
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9712:
URL: https://github.com/apache/pinot/pull/9712#issuecomment-1301500005
# [Codecov](https://codecov.io/gh/apache/pinot/pull/9712?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9712](https://codecov.io/gh/apache/pinot/pull/9712?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (31ca8aa) into [master](https://codecov.io/gh/apache/pinot/commit/91fa86b2a30e1d7ec0ac57f5ce4e22b42ad5cb4e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (91fa86b) will **increase** coverage by `39.40%`.
> The diff coverage is `85.36%`.
```diff
@@ Coverage Diff @@
## master #9712 +/- ##
=============================================
+ Coverage 28.13% 67.53% +39.40%
- Complexity 53 5162 +5109
=============================================
Files 1936 1448 -488
Lines 103927 75791 -28136
Branches 15770 12073 -3697
=============================================
+ Hits 29235 51182 +21947
+ Misses 71832 20947 -50885
- Partials 2860 3662 +802
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `67.53% <85.36%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9712?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...a/org/apache/pinot/segment/spi/ColumnMetadata.java](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0NvbHVtbk1ldGFkYXRhLmphdmE=) | `80.00% <ø> (+80.00%)` | :arrow_up: |
| [...segment/spi/index/metadata/ColumnMetadataImpl.java](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L21ldGFkYXRhL0NvbHVtbk1ldGFkYXRhSW1wbC5qYXZh) | `88.75% <66.66%> (+88.75%)` | :arrow_up: |
| [...ache/pinot/segment/spi/store/ColumnIndexUtils.java](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL0NvbHVtbkluZGV4VXRpbHMuamF2YQ==) | `77.77% <77.77%> (ø)` | |
| [...t/query/runtime/plan/ServerRequestPlanVisitor.java](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9wbGFuL1NlcnZlclJlcXVlc3RQbGFuVmlzaXRvci5qYXZh) | `78.57% <78.57%> (ø)` | |
| [.../local/segment/store/SingleFileIndexDirectory.java](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3N0b3JlL1NpbmdsZUZpbGVJbmRleERpcmVjdG9yeS5qYXZh) | `90.72% <83.33%> (+90.72%)` | :arrow_up: |
| [...va/org/apache/pinot/query/runtime/QueryRunner.java](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9RdWVyeVJ1bm5lci5qYXZh) | `81.55% <90.00%> (+81.55%)` | :arrow_up: |
| [...e/pinot/query/runtime/plan/PlanRequestContext.java](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9wbGFuL1BsYW5SZXF1ZXN0Q29udGV4dC5qYXZh) | `92.85% <92.85%> (ø)` | |
| [...ot/query/runtime/executor/WorkerQueryExecutor.java](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9leGVjdXRvci9Xb3JrZXJRdWVyeUV4ZWN1dG9yLmphdmE=) | `100.00% <100.00%> (+100.00%)` | :arrow_up: |
| [.../pinot/query/runtime/plan/PhysicalPlanVisitor.java](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9wbGFuL1BoeXNpY2FsUGxhblZpc2l0b3IuamF2YQ==) | `96.87% <100.00%> (ø)` | |
| [.../runtime/plan/server/ServerPlanRequestContext.java](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9wbGFuL3NlcnZlci9TZXJ2ZXJQbGFuUmVxdWVzdENvbnRleHQuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| ... and [1702 more](https://codecov.io/gh/apache/pinot/pull/9712/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr merged pull request #9712: add index key and size map to segment metadata
Posted by GitBox <gi...@apache.org>.
walterddr merged PR #9712:
URL: https://github.com/apache/pinot/pull/9712
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on a diff in pull request #9712: add index key and size map to segment metadata
Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #9712:
URL: https://github.com/apache/pinot/pull/9712#discussion_r1014316651
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/SegmentMetadataImpl.java:
##########
@@ -245,6 +248,26 @@ private void init(PropertiesConfiguration segmentMetadataPropertiesConfiguration
_schema.addField(columnMetadata.getFieldSpec());
}
+ // Load index metadata
+ // Support V3 (e.g. SingleFileIndexDirectory only)
+ if (_segmentVersion == SegmentVersion.v3) {
+ File indexMapFile = new File(_indexDir, "v3" + File.separator + V1Constants.INDEX_MAP_FILE_NAME);
+ if (indexMapFile.exists()) {
+ PropertiesConfiguration mapConfig = CommonsConfigurationUtils.fromFile(indexMapFile);
+ for (String key : CommonsConfigurationUtils.getKeys(mapConfig)) {
+ try {
+ String[] parsedKeys = ColumnIndexUtils.parseIndexMapKeys(key, _indexDir.getPath());
+ if (parsedKeys[2].equals(ColumnIndexUtils.MAP_KEY_NAME_SIZE)) {
+ ColumnIndexType columnIndexType = ColumnIndexType.getValue(parsedKeys[1]);
+ _columnMetadataMap.get(parsedKeys[0]).getIndexSizeMap().put(columnIndexType, mapConfig.getLong(key));
Review Comment:
ah missed that 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9712: add index key and size map to segment metadata
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9712:
URL: https://github.com/apache/pinot/pull/9712#discussion_r1014314067
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/SegmentMetadataImpl.java:
##########
@@ -245,6 +248,26 @@ private void init(PropertiesConfiguration segmentMetadataPropertiesConfiguration
_schema.addField(columnMetadata.getFieldSpec());
}
+ // Load index metadata
+ // Support V3 (e.g. SingleFileIndexDirectory only)
+ if (_segmentVersion == SegmentVersion.v3) {
+ File indexMapFile = new File(_indexDir, "v3" + File.separator + V1Constants.INDEX_MAP_FILE_NAME);
+ if (indexMapFile.exists()) {
+ PropertiesConfiguration mapConfig = CommonsConfigurationUtils.fromFile(indexMapFile);
+ for (String key : CommonsConfigurationUtils.getKeys(mapConfig)) {
+ try {
+ String[] parsedKeys = ColumnIndexUtils.parseIndexMapKeys(key, _indexDir.getPath());
+ if (parsedKeys[2].equals(ColumnIndexUtils.MAP_KEY_NAME_SIZE)) {
+ ColumnIndexType columnIndexType = ColumnIndexType.getValue(parsedKeys[1]);
+ _columnMetadataMap.get(parsedKeys[0]).getIndexSizeMap().put(columnIndexType, mapConfig.getLong(key));
Review Comment:
i believe it is already caught in the Exception below?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9712: add index key and size map to segment metadata
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9712:
URL: https://github.com/apache/pinot/pull/9712#discussion_r1012373415
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -65,13 +66,14 @@ public class ColumnMetadataImpl implements ColumnMetadata {
private final int _totalNumberOfEntries;
private final PartitionFunction _partitionFunction;
private final Set<Integer> _partitions;
+ private final Map<ColumnIndexType, Long> _indexSizeMap;
Review Comment:
resolved. no need for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9712: add index key and size map to segment metadata
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9712:
URL: https://github.com/apache/pinot/pull/9712#discussion_r1012329579
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -65,13 +66,14 @@ public class ColumnMetadataImpl implements ColumnMetadata {
private final int _totalNumberOfEntries;
private final PartitionFunction _partitionFunction;
private final Set<Integer> _partitions;
+ private final Map<ColumnIndexType, Long> _indexSizeMap;
Review Comment:
planning to move IndexKey or IndexEntry (without the buffer) from segment-local to segment-spi. thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9712: add index key and size map to segment metadata
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9712:
URL: https://github.com/apache/pinot/pull/9712#discussion_r1012324991
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/SegmentMetadataImpl.java:
##########
@@ -209,7 +216,8 @@ private void loadCreationMeta(InputStream crcFileInputStream)
}
}
- private void init(PropertiesConfiguration segmentMetadataPropertiesConfiguration) {
+ private void init(PropertiesConfiguration segmentMetadataPropertiesConfiguration,
+ PropertiesConfiguration segmentIndexPropertiesConfiguration) {
Review Comment:
no implemented yet: still need to parse this index PropertiesConfiguration but it should have everything
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on a diff in pull request #9712: add index key and size map to segment metadata
Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #9712:
URL: https://github.com/apache/pinot/pull/9712#discussion_r1014306434
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/SegmentMetadataImpl.java:
##########
@@ -245,6 +248,26 @@ private void init(PropertiesConfiguration segmentMetadataPropertiesConfiguration
_schema.addField(columnMetadata.getFieldSpec());
}
+ // Load index metadata
+ // Support V3 (e.g. SingleFileIndexDirectory only)
+ if (_segmentVersion == SegmentVersion.v3) {
+ File indexMapFile = new File(_indexDir, "v3" + File.separator + V1Constants.INDEX_MAP_FILE_NAME);
+ if (indexMapFile.exists()) {
+ PropertiesConfiguration mapConfig = CommonsConfigurationUtils.fromFile(indexMapFile);
+ for (String key : CommonsConfigurationUtils.getKeys(mapConfig)) {
+ try {
+ String[] parsedKeys = ColumnIndexUtils.parseIndexMapKeys(key, _indexDir.getPath());
+ if (parsedKeys[2].equals(ColumnIndexUtils.MAP_KEY_NAME_SIZE)) {
+ ColumnIndexType columnIndexType = ColumnIndexType.getValue(parsedKeys[1]);
+ _columnMetadataMap.get(parsedKeys[0]).getIndexSizeMap().put(columnIndexType, mapConfig.getLong(key));
Review Comment:
do we need a nullcheck on the value of get() here? could it be that something is in index_map but not in metadata.properties (due to a column remove/index remove)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org