You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/04/27 02:29:38 UTC
[GitHub] [skywalking] lngmountain opened a new pull request, #8958: fix bug: Even after a rerun of init mode job, TimeSeriesTable ES index not created.
lngmountain opened a new pull request, #8958:
URL: https://github.com/apache/skywalking/pull/8958
Fix the problem that es index(TimeSeriesTable, eg. endpoint_traffic, alarm_record) didn't created even after rerun with init-mode. This problem caused the oap server fail to start when the oap server down for more than a day.
- [] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [X] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/changelog/docs/en/changes/changes.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #8958: fix bug: Even after a rerun of init mode job, TimeSeriesTable ES index not created.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #8958:
URL: https://github.com/apache/skywalking/pull/8958#discussion_r859446262
##########
docs/en/changes/changes.md:
##########
@@ -28,6 +28,7 @@
* Fix the problem that some configurations (such as group.id) did not take effect due to the override order when using the kafkaConsumerConfig property to extend the configuration in Kafka Fetcher.
* Remove build time from the OAP version.
* Add data-generator module to run OAP in testing mode, generating mock data for testing.
+* Fix the problem that es index(TimeSeriesTable, eg. endpoint_traffic, alarm_record) didn't created even after rerun with init-mode. This problem caused the oap server fail to start when the oap server down for more than a day.
Review Comment:
```suggestion
* Fix the problem that es index(TimeSeriesTable, eg. endpoint_traffic, alarm_record) didn't create even after rerun with init-mode. This problem caused the OAP server to fail to start when the OAP server was down for more than a day.
```
Fix grammar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] lngmountain commented on pull request #8958: fix bug: Even after a rerun of init mode job, TimeSeriesTable ES index not created.
Posted by GitBox <gi...@apache.org>.
lngmountain commented on PR #8958:
URL: https://github.com/apache/skywalking/pull/8958#issuecomment-1111040121
Let me describe the problem I encountered and how to solve it.
My OAP server was down form On April 25 to today, I got `table: endpoint_traffic does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later` when I tried to start OAP server. Then I try to rerun init-mode job and got the same log `table: endpoint_traffic does not exist`, but the es index didn't created.
I compared the following two pieces of code in class StorageEsInstaller,
`@Override
protected boolean isExists(Model model) {
ElasticSearchClient esClient = (ElasticSearchClient) client;
String tableName = IndexController.INSTANCE.getTableName(model);
IndexController.LogicIndicesRegister.registerRelation(model.getName(), tableName);
if (!model.isTimeSeries()) {
boolean exist = esClient.isExistsIndex(tableName);
if (exist) {
Mappings historyMapping = esClient.getIndex(tableName)
.map(Index::getMappings)
.orElseGet(Mappings::new);
structures.putStructure(tableName, historyMapping);
exist = structures.containsStructure(tableName, createMapping(model));
}
return exist;
}
boolean templateExists = esClient.isExistsTemplate(tableName);
final Optional<IndexTemplate> template = esClient.getTemplate(tableName);
boolean lastIndexExists = esClient.isExistsIndex(TimeSeriesUtils.latestWriteIndexName(model));
if ((templateExists && !template.isPresent()) || (!templateExists && template.isPresent())) {
throw new Error("[Bug warning] ElasticSearch client query template result is not consistent. " +
"Please file an issue to Apache SkyWalking.(https://github.com/apache/skywalking/issues)");
}
boolean exist = templateExists && lastIndexExists;
if (exist) {
structures.putStructure(
tableName, template.get().getMappings()
);
exist = structures.containsStructure(tableName, createMapping(model));
}
return exist;
}`
The code above that execute on OAP startup shows that the model exists only if both `template` and `lastIndex` exist, but the following code that execute on model not exist shows that `template` and `lastIndex` are created only if `template` not exists, these two judgments do not match and lastIndex will not be created.
`private void createTimeSeriesTable(Model model) throws StorageException {
ElasticSearchClient esClient = (ElasticSearchClient) client;
String tableName = IndexController.INSTANCE.getTableName(model);
Map<String, Object> settings = createSetting(model);
Mappings mapping = createMapping(model);
String indexName = TimeSeriesUtils.latestWriteIndexName(model);
try {
boolean shouldUpdateTemplate = !esClient.isExistsTemplate(tableName);
shouldUpdateTemplate = shouldUpdateTemplate || !structures.containsStructure(tableName, mapping);
if (shouldUpdateTemplate) {
structures.putStructure(tableName, mapping);
boolean isAcknowledged = esClient.createOrUpdateTemplate(
tableName, settings, structures.getMapping(tableName), config.getIndexTemplateOrder());
log.info("create {} index template finished, isAcknowledged: {}", tableName, isAcknowledged);
if (!isAcknowledged) {
throw new IOException("create " + tableName + " index template failure, ");
}
if (esClient.isExistsIndex(indexName)) {
Mappings historyMapping = esClient.getIndex(indexName)
.map(Index::getMappings)
.orElseGet(Mappings::new);
Mappings appendMapping = structures.diffStructure(tableName, historyMapping);
if (appendMapping.getProperties() != null && !appendMapping.getProperties().isEmpty()) {
isAcknowledged = esClient.updateIndexMapping(indexName, appendMapping);
log.info("update {} index finished, isAcknowledged: {}, append mappings: {}", indexName,
isAcknowledged, appendMapping
);
if (!isAcknowledged) {
throw new StorageException("update " + indexName + " time series index failure");
}
}
} else {
isAcknowledged = esClient.createIndex(indexName);
log.info("create {} index finished, isAcknowledged: {}", indexName, isAcknowledged);
if (!isAcknowledged) {
throw new StorageException("create " + indexName + " time series index failure");
}
}
}
} catch (IOException e) {
throw new StorageException("cannot create " + tableName + " index template", e);
}
}`
The existence of `template` does not imply the existence of model's `lastIndex`, so `template` and `lastIndex` should be handled independently in the `createTimeSeriesTable` method. I tuned the code this way and fixed the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] lngmountain commented on pull request #8958: fix bug: Even after a rerun of init mode job, TimeSeriesTable ES index not created.
Posted by GitBox <gi...@apache.org>.
lngmountain commented on PR #8958:
URL: https://github.com/apache/skywalking/pull/8958#issuecomment-1110486420
> Could you please follow the pr template to add some summaries to explain why this bug happened and how to fix it? We would appreciate it very much.
OK，I'll give some information later today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng merged pull request #8958: fix bug: Even after a rerun of init mode job, TimeSeriesTable ES index not created.
Posted by GitBox <gi...@apache.org>.
wu-sheng merged PR #8958:
URL: https://github.com/apache/skywalking/pull/8958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8958: fix bug: Even after a rerun of init mode job, TimeSeriesTable ES index not created.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8958:
URL: https://github.com/apache/skywalking/pull/8958#issuecomment-1111045008
@lngmountain A tip, ``` is a recommended markdown format to quote codes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wankai123 commented on pull request #8958: fix bug: Even after a rerun of init mode job, TimeSeriesTable ES index not created.
Posted by GitBox <gi...@apache.org>.
wankai123 commented on PR #8958:
URL: https://github.com/apache/skywalking/pull/8958#issuecomment-1110477563
Could you please follow the pr template to add some summaries to explain why this bug happened and how to fix it? We would appreciate it very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8958: fix bug: Even after a rerun of init mode job, TimeSeriesTable ES index not created.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8958:
URL: https://github.com/apache/skywalking/pull/8958#issuecomment-1110958128
@lngmountain Anything to update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org