You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/05/17 20:50:05 UTC
[GitHub] [pinot] Jackie-Jiang opened a new pull request, #8721: Fix dimension table load when server restart or reload table
Jackie-Jiang opened a new pull request, #8721:
URL: https://github.com/apache/pinot/pull/8721
Currently `DimensionTableDataManager` overrides the `addSegment(File indexDir, IndexLoadingConfig indexLoadingConfig)` method to reload the lookup table, but this method is not always invoked when adding the segment. Fix the issue by overriding `addSegment(ImmutableSegment immutableSegment)` which is always invoked when adding the segment.
Other changes:
- Enhance the validation for schema and primary key and fail with proper exception message
- Enhance the test to mimic the actual flow of reloading the segment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #8721: Fix dimension table load when server restart or reload table
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged PR #8721:
URL: https://github.com/apache/pinot/pull/8721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] navina commented on a diff in pull request #8721: Fix dimension table load when server restart or reload table
Posted by GitBox <gi...@apache.org>.
navina commented on code in PR #8721:
URL: https://github.com/apache/pinot/pull/8721#discussion_r875355439
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java:
##########
@@ -128,25 +128,26 @@ private void loadLookupTable()
} while (!UPDATER.compareAndSet(this, snapshot, replacement));
}
- private DimensionTable createDimensionTable()
- throws Exception {
- Map<PrimaryKey, GenericRow> map = new HashMap<>();
+ private DimensionTable createDimensionTable() {
+ Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore, _tableNameWithType);
+ Preconditions.checkState(schema != null, "Failed to find schema for dimension table: %s", _tableNameWithType);
+ List<String> primaryKeyColumns = schema.getPrimaryKeyColumns();
+ Preconditions.checkState(CollectionUtils.isNotEmpty(primaryKeyColumns),
+ "Primary key columns must be configured for dimension table: %s", _tableNameWithType);
+
+ Map<PrimaryKey, GenericRow> lookupTable = new HashMap<>();
List<SegmentDataManager> segmentManagers = acquireAllSegments();
try {
for (SegmentDataManager segmentManager : segmentManagers) {
IndexSegment indexSegment = segmentManager.getSegment();
- try (PinotSegmentRecordReader reader = new PinotSegmentRecordReader(
- indexSegment.getSegmentMetadata().getIndexDir())) {
- while (reader.hasNext()) {
- GenericRow row = reader.next();
- map.put(row.getPrimaryKey(_dimensionTable.getPrimaryKeyColumns()), row);
- }
+ int numTotalDocs = indexSegment.getSegmentMetadata().getTotalDocs();
Review Comment:
Makes sense. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8721: Fix dimension table load when server restart or reload table
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8721:
URL: https://github.com/apache/pinot/pull/8721#discussion_r875303678
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java:
##########
@@ -128,25 +128,26 @@ private void loadLookupTable()
} while (!UPDATER.compareAndSet(this, snapshot, replacement));
}
- private DimensionTable createDimensionTable()
- throws Exception {
- Map<PrimaryKey, GenericRow> map = new HashMap<>();
+ private DimensionTable createDimensionTable() {
+ Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore, _tableNameWithType);
+ Preconditions.checkState(schema != null, "Failed to find schema for dimension table: %s", _tableNameWithType);
+ List<String> primaryKeyColumns = schema.getPrimaryKeyColumns();
+ Preconditions.checkState(CollectionUtils.isNotEmpty(primaryKeyColumns),
+ "Primary key columns must be configured for dimension table: %s", _tableNameWithType);
+
+ Map<PrimaryKey, GenericRow> lookupTable = new HashMap<>();
List<SegmentDataManager> segmentManagers = acquireAllSegments();
try {
for (SegmentDataManager segmentManager : segmentManagers) {
IndexSegment indexSegment = segmentManager.getSegment();
- try (PinotSegmentRecordReader reader = new PinotSegmentRecordReader(
- indexSegment.getSegmentMetadata().getIndexDir())) {
- while (reader.hasNext()) {
- GenericRow row = reader.next();
- map.put(row.getPrimaryKey(_dimensionTable.getPrimaryKeyColumns()), row);
- }
+ int numTotalDocs = indexSegment.getSegmentMetadata().getTotalDocs();
Review Comment:
Good question. We don't want to create a separate record reader because that will load the segment again. Instead, we can directly read the record out of the already loaded segment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #8721: Fix dimension table load when server restart or reload table
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8721:
URL: https://github.com/apache/pinot/pull/8721#issuecomment-1129348268
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8721](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c84901) into [master](https://codecov.io/gh/apache/pinot/commit/3bbae6ef6145ca29af7f8d0f115263437eef63bd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3bbae6e) will **increase** coverage by `3.41%`.
> The diff coverage is `76.00%`.
```diff
@@ Coverage Diff @@
## master #8721 +/- ##
============================================
+ Coverage 66.20% 69.62% +3.41%
- Complexity 4389 4576 +187
============================================
Files 1279 1729 +450
Lines 64619 90183 +25564
Branches 10089 13415 +3326
============================================
+ Hits 42781 62786 +20005
- Misses 18802 23044 +4242
- Partials 3036 4353 +1317
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `26.96% <0.00%> (?)` | |
| integration2 | `25.34% <0.00%> (?)` | |
| unittests1 | `66.13% <76.00%> (-0.07%)` | :arrow_down: |
| unittests2 | `14.27% <0.00%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/offline/DimensionTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9EaW1lbnNpb25UYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `82.14% <76.00%> (-2.48%)` | :arrow_down: |
| [...ore/query/scheduler/resources/ResourceManager.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvcmVzb3VyY2VzL1Jlc291cmNlTWFuYWdlci5qYXZh) | `84.00% <0.00%> (-12.00%)` | :arrow_down: |
| [.../impl/dictionary/BaseOffHeapMutableDictionary.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvQmFzZU9mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh) | `81.33% <0.00%> (-3.34%)` | :arrow_down: |
| [.../pinot/core/query/scheduler/PriorityScheduler.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvUHJpb3JpdHlTY2hlZHVsZXIuamF2YQ==) | `78.08% <0.00%> (-2.74%)` | :arrow_down: |
| [...e/pinot/segment/local/io/util/PinotDataBitSet.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pby91dGlsL1Bpbm90RGF0YUJpdFNldC5qYXZh) | `95.62% <0.00%> (-1.46%)` | :arrow_down: |
| [...org/apache/pinot/core/common/ObjectSerDeUtils.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vT2JqZWN0U2VyRGVVdGlscy5qYXZh) | `89.57% <0.00%> (-0.82%)` | :arrow_down: |
| [.../aggregation/function/ModeAggregationFunction.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9Nb2RlQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `87.56% <0.00%> (-0.82%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `12.70% <0.00%> (-0.22%)` | :arrow_down: |
| [...ingestion/batch/spec/SegmentGenerationJobSpec.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvaW5nZXN0aW9uL2JhdGNoL3NwZWMvU2VnbWVudEdlbmVyYXRpb25Kb2JTcGVjLmphdmE=) | `53.33% <0.00%> (ø)` | |
| [...ent/local/data/manager/TableDataManagerConfig.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9kYXRhL21hbmFnZXIvVGFibGVEYXRhTWFuYWdlckNvbmZpZy5qYXZh) | `0.00% <0.00%> (ø)` | |
| ... and [688 more](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3bbae6e...5c84901](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] navina commented on a diff in pull request #8721: Fix dimension table load when server restart or reload table
Posted by GitBox <gi...@apache.org>.
navina commented on code in PR #8721:
URL: https://github.com/apache/pinot/pull/8721#discussion_r875264972
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java:
##########
@@ -128,25 +128,26 @@ private void loadLookupTable()
} while (!UPDATER.compareAndSet(this, snapshot, replacement));
}
- private DimensionTable createDimensionTable()
- throws Exception {
- Map<PrimaryKey, GenericRow> map = new HashMap<>();
+ private DimensionTable createDimensionTable() {
+ Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore, _tableNameWithType);
+ Preconditions.checkState(schema != null, "Failed to find schema for dimension table: %s", _tableNameWithType);
+ List<String> primaryKeyColumns = schema.getPrimaryKeyColumns();
+ Preconditions.checkState(CollectionUtils.isNotEmpty(primaryKeyColumns),
+ "Primary key columns must be configured for dimension table: %s", _tableNameWithType);
+
+ Map<PrimaryKey, GenericRow> lookupTable = new HashMap<>();
List<SegmentDataManager> segmentManagers = acquireAllSegments();
try {
for (SegmentDataManager segmentManager : segmentManagers) {
IndexSegment indexSegment = segmentManager.getSegment();
- try (PinotSegmentRecordReader reader = new PinotSegmentRecordReader(
- indexSegment.getSegmentMetadata().getIndexDir())) {
- while (reader.hasNext()) {
- GenericRow row = reader.next();
- map.put(row.getPrimaryKey(_dimensionTable.getPrimaryKeyColumns()), row);
- }
+ int numTotalDocs = indexSegment.getSegmentMetadata().getTotalDocs();
Review Comment:
qq: Why not continue using the `PinotSegmentRecordReader` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org