You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2021/09/01 13:38:13 UTC
[GitHub] [skywalking] kezhenxu94 opened a new pull request #7634: Rebuilt ElasticSearch client on top of their REST API
kezhenxu94 opened a new pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634
<!--
โ ๏ธ Please make sure to read this template first, pull requests that don't accord with this template
maybe closed without notice.
Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
-->
<!-- ==== ๐ Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist ๐ ====
### Fix <bug description or the bug issue number or bug issue link>
- [ ] Add a unit test to verify that the fix works.
- [ ] Explain briefly why the bug exists and how to fix it.
==== ๐ Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist ๐ ==== -->
<!-- ==== ๐ Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist ๐ ====
### Improve the performance of <class or module or ...>
- [ ] Add a benchmark for the improvement, refer to [the existing ones](https://github.com/apache/skywalking/blob/master/apm-commons/apm-datacarrier/src/test/java/org/apache/skywalking/apm/commons/datacarrier/LinkedArrayBenchmark.java)
- [ ] The benchmark result.
```text
<Paste the benchmark results here>
```
- [ ] Links/URLs to the theory proof or discussion articles/blogs. <links/URLs here>
==== ๐ Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist ๐ ==== -->
<!-- ==== ๐ Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist ๐ ====
### <Feature description>
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [ ] Update the documentation to include this new feature.
- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
- [ ] If it's UI related, attach the screenshots below.
==== ๐ Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist ๐ ==== -->
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/CHANGES.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913381194
> `docker/oap/Dockerfile.oap` will be fixed in follow-up PRs?
I think all needed changes to docker (and `docker/oap/Dockerfile.oap`) are included in this PR, the official docker images will be done in http://github.com/apache/skywalking-docker after this PR, if I missed anything, please let me know @hanahmily
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702418051
##########
File path: docs/en/setup/backend/backend-storage.md
##########
@@ -34,23 +34,18 @@ storage:
## OpenSearch
-OpenSearch storage shares the same configurations as ElasticSearch 7.
-In order to activate ElasticSearch 7 as storage, set storage provider to **elasticsearch7**.
-Please download the `apache-skywalking-bin-es7.tar.gz` if you want to use OpenSearch as storage.
+OpenSearch storage shares the same configurations as ElasticSearch.
+In order to activate OpenSearch as storage, set storage provider to **elasticsearch**.
## ElasticSearch
**NOTE:** Elastic announced through their blog that Elasticsearch will be moving over to a Server Side Public
License (SSPL), which is incompatible with Apache License 2.0. This license change is effective from Elasticsearch
version 7.11. So please choose the suitable ElasticSearch version according to your usage.
-- In order to activate ElasticSearch 6 as storage, set storage provider to **elasticsearch**
-- In order to activate ElasticSearch 7 as storage, set storage provider to **elasticsearch7**
-
-**Required ElasticSearch 6.3.2 or higher. HTTP RestHighLevelClient is used to connect server.**
-
-- For ElasticSearch 6.3.2 ~ 7.0.0 (excluded), please download `apache-skywalking-bin.tar.gz`.
-- For ElasticSearch 7.0.0 ~ 8.0.0 (excluded), please download `apache-skywalking-bin-es7.tar.gz`.
+Since 8.8.0, SkyWalking rebuilds the ElasticSearch client on top of ElasticSearch REST API and automatically picks up
+correct request formats according to the server side version, hence you don't need to download different binaries
+and don't need to configure different storage selector for different ElasticSearch server side version anymore.
Review comment:
I think we still need to indicate the supported-versions range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 edited a comment on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 edited a comment on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913381194
> `docker/oap/Dockerfile.oap` will be fixed in follow-up PRs?
~~I think all needed changes to docker (and `docker/oap/Dockerfile.oap`) are included in this PR, the official docker images will be done in http://github.com/apache/skywalking-docker after this PR, if I missed anything, please let me know @hanahmily ~~
ignore this too ๐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
hanahmily commented on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913381046
> `docker/oap/Dockerfile.oap` will be fixed in follow-up PRs?
ignore this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913286498
Now the PR should be good to go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702409042
##########
File path: oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/ElasticSearchVersion.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.skywalking.library.elasticsearch;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.apache.skywalking.library.elasticsearch.requests.factory.Codec;
+import org.apache.skywalking.library.elasticsearch.requests.factory.RequestFactory;
+import org.apache.skywalking.library.elasticsearch.requests.factory.v6.V6RequestFactory;
+import org.apache.skywalking.library.elasticsearch.requests.factory.v6.codec.V6Codec;
+import org.apache.skywalking.library.elasticsearch.requests.factory.v7.V78RequestFactory;
+import org.apache.skywalking.library.elasticsearch.requests.factory.v7.V7RequestFactory;
+import org.apache.skywalking.library.elasticsearch.requests.factory.v7.codec.V7Codec;
+
+public final class ElasticSearchVersion {
+ private final String distribution;
+ private final int major;
+ private final int minor;
+
+ private final RequestFactory requestFactory;
+ private final Codec codec;
+
+ private ElasticSearchVersion(final String distribution, final int major, final int minor) {
+ this.distribution = distribution;
+ this.major = major;
+ this.minor = minor;
+
+ if (distribution.equalsIgnoreCase("OpenSearch")) {
+ requestFactory = new V78RequestFactory(this);
+ codec = V7Codec.INSTANCE;
+ return;
+ }
+
+ if (distribution.equalsIgnoreCase("ElasticSearch")) {
+ if (major == 6) { // 6.x
+ requestFactory = new V6RequestFactory(this);
+ codec = V6Codec.INSTANCE;
+ return;
+ }
+ if (major == 7) {
+ codec = V7Codec.INSTANCE;
+ if (minor < 8) { // [7.0, 7.8)
+ requestFactory = new V7RequestFactory(this);
+ } else { // [7.8, 8.0)
Review comment:
> Are We going to test v8 in this PR?
No, this PR is already large enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913148598
I haven't reviewed the client's implementations, as you mentioned `TODO` should be fixed later. But besides that, this is generally good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702418173
##########
File path: oap-server/server-configuration/configuration-apollo/pom.xml
##########
@@ -54,6 +54,11 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <scope>test</scope>
+ </dependency>
Review comment:
Why do we need to add a test dependency but no code(test code) change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702398273
##########
File path: oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/ElasticSearchVersion.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.skywalking.library.elasticsearch;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.apache.skywalking.library.elasticsearch.requests.factory.Codec;
+import org.apache.skywalking.library.elasticsearch.requests.factory.RequestFactory;
+import org.apache.skywalking.library.elasticsearch.requests.factory.v6.V6RequestFactory;
+import org.apache.skywalking.library.elasticsearch.requests.factory.v6.codec.V6Codec;
+import org.apache.skywalking.library.elasticsearch.requests.factory.v7.V78RequestFactory;
+import org.apache.skywalking.library.elasticsearch.requests.factory.v7.V7RequestFactory;
+import org.apache.skywalking.library.elasticsearch.requests.factory.v7.codec.V7Codec;
+
+public final class ElasticSearchVersion {
+ private final String distribution;
+ private final int major;
+ private final int minor;
+
+ private final RequestFactory requestFactory;
+ private final Codec codec;
+
+ private ElasticSearchVersion(final String distribution, final int major, final int minor) {
+ this.distribution = distribution;
+ this.major = major;
+ this.minor = minor;
+
+ if (distribution.equalsIgnoreCase("OpenSearch")) {
+ requestFactory = new V78RequestFactory(this);
+ codec = V7Codec.INSTANCE;
+ return;
+ }
+
+ if (distribution.equalsIgnoreCase("ElasticSearch")) {
+ if (major == 6) { // 6.x
+ requestFactory = new V6RequestFactory(this);
+ codec = V6Codec.INSTANCE;
+ return;
+ }
+ if (major == 7) {
+ codec = V7Codec.INSTANCE;
+ if (minor < 8) { // [7.0, 7.8)
+ requestFactory = new V7RequestFactory(this);
+ } else { // [7.8, 8.0)
Review comment:
Are We going to test v8 in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702417862
##########
File path: oap-server/server-starter/src/main/java/org/apache/skywalking/oap/server/starter/OAPServerStartUp.java
##########
@@ -18,9 +18,6 @@
package org.apache.skywalking.oap.server.starter;
-/**
- * OAP starter
- */
public class OAPServerStartUp {
Review comment:
Considering ES7 starter removed, do you still need to keep this module? How about moving this into bootstrap directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 merged pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 merged pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702549697
##########
File path: docs/en/setup/backend/configuration-vocabulary.md
##########
@@ -79,12 +79,13 @@ core|default|role|Option values: `Mixed/Receiver/Aggregator`. **Receiver** mode
| - | - | password | Nacos Auth password. | SW_CLUSTER_NACOS_PASSWORD | - |
| - | - | accessKey | Nacos Auth accessKey. | SW_CLUSTER_NACOS_ACCESSKEY | - |
| - | - | secretKey | Nacos Auth secretKey. | SW_CLUSTER_NACOS_SECRETKEY | - |
-| storage|elasticsearch| - | ElasticSearch 6 storage implementation. | - | - |
+| storage|elasticsearch| - | ElasticSearch (and OpenSearch) storage implementation. | - | - |
| - | - | nameSpace | Prefix of indexes created and used by SkyWalking. | SW_NAMESPACE | - |
| - | - | clusterNodes | ElasticSearch cluster nodes for client connection.| SW_STORAGE_ES_CLUSTER_NODES |localhost|
| - | - | protocol | HTTP or HTTPs. | SW_STORAGE_ES_HTTP_PROTOCOL | HTTP|
| - | - | connectTimeout | Connect timeout of ElasticSearch client (in milliseconds). | SW_STORAGE_ES_CONNECT_TIMEOUT | 500|
| - | - | socketTimeout | Socket timeout of ElasticSearch client (in milliseconds). | SW_STORAGE_ES_SOCKET_TIMEOUT | 30000|
+| - | - | numHttpClientThread | The number of threads for the underlying HTTP client to perform socket I/O. If the value is <= 0, the number of available processors will be used. | SW_STORAGE_ES_NUM_HTTP_CLIENT_THREAD | 0 |
Review comment:
Just one check, all metrics/log/traces persistent are according to bulk thread today, right? So, it always uses 2 threads in default. Then others are for query if there are over 2 cores.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702399090
##########
File path: oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/bulk/BulkProcessor.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.skywalking.library.elasticsearch.bulk;
+
+import com.linecorp.armeria.client.WebClient;
+import com.linecorp.armeria.common.HttpStatus;
+import com.linecorp.armeria.common.util.Exceptions;
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.library.elasticsearch.ElasticSearchVersion;
+import org.apache.skywalking.library.elasticsearch.requests.IndexRequest;
+import org.apache.skywalking.library.elasticsearch.requests.UpdateRequest;
+import org.apache.skywalking.library.elasticsearch.requests.factory.RequestFactory;
+
+import static java.util.Objects.requireNonNull;
+
+@Slf4j
+public final class BulkProcessor {
+ private final ArrayBlockingQueue<Object> requests;
+
+ private final CompletableFuture<ElasticSearchVersion> version;
+ private final WebClient client;
+
+ private final int bulkActions;
+ private final int concurrentRequests;
+
+ public BulkProcessor(
+ final CompletableFuture<ElasticSearchVersion> version,
+ final WebClient client, final int bulkActions,
+ final Duration flushInterval, final int concurrentRequests) {
+ this.version = requireNonNull(version, "requestFactory");
+ this.client = requireNonNull(client, "client");
+ this.bulkActions = bulkActions;
+
+ requireNonNull(flushInterval, "flushInterval");
+
+ this.concurrentRequests = concurrentRequests;
+ this.requests = new ArrayBlockingQueue<>(bulkActions + 1);
+ Executors.newSingleThreadScheduledExecutor(r -> {
Review comment:
One thread only? It seems `concurrentRequests` not being used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 edited a comment on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 edited a comment on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913381194
> `docker/oap/Dockerfile.oap` will be fixed in follow-up PRs?
~~I think all needed changes to docker (and `docker/oap/Dockerfile.oap`) are included in this PR, the official docker images will be done in http://github.com/apache/skywalking-docker after this PR, if I missed anything, please let me know @hanahmily~~
ignore this too ๐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702397591
##########
File path: oap-server/server-library/library-elasticsearch-client/pom.xml
##########
@@ -0,0 +1,59 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-library</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>library-elasticsearch-client</artifactId>
+ <description>
+ An ElasticSearch client built on top of ElasticSearch REST API.
+ Note that this client only covers part of the features that are used in SkyWalking,
+ and it's not for general use.
+ </description>
Review comment:
Good to have this. FYI @apache/skywalking-committers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702418938
##########
File path: oap-server/server-configuration/configuration-apollo/pom.xml
##########
@@ -54,6 +54,11 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ <scope>test</scope>
+ </dependency>
Review comment:
> Why do we need to add a test dependency but no code(test code) change?
This dependency was brought by ElasticSearch HLRC in `library-client` (as runtime dependency), since ES HLRC is removed, we have to add it explicitly, the same for other modules dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913110569
Mark it as ready for review for reviewers who want to jump in as earlier as possible, though there are a lot "TODO" and to be optimized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702524474
##########
File path: docs/en/setup/backend/configuration-vocabulary.md
##########
@@ -79,12 +79,13 @@ core|default|role|Option values: `Mixed/Receiver/Aggregator`. **Receiver** mode
| - | - | password | Nacos Auth password. | SW_CLUSTER_NACOS_PASSWORD | - |
| - | - | accessKey | Nacos Auth accessKey. | SW_CLUSTER_NACOS_ACCESSKEY | - |
| - | - | secretKey | Nacos Auth secretKey. | SW_CLUSTER_NACOS_SECRETKEY | - |
-| storage|elasticsearch| - | ElasticSearch 6 storage implementation. | - | - |
+| storage|elasticsearch| - | ElasticSearch (and OpenSearch) storage implementation. | - | - |
| - | - | nameSpace | Prefix of indexes created and used by SkyWalking. | SW_NAMESPACE | - |
| - | - | clusterNodes | ElasticSearch cluster nodes for client connection.| SW_STORAGE_ES_CLUSTER_NODES |localhost|
| - | - | protocol | HTTP or HTTPs. | SW_STORAGE_ES_HTTP_PROTOCOL | HTTP|
| - | - | connectTimeout | Connect timeout of ElasticSearch client (in milliseconds). | SW_STORAGE_ES_CONNECT_TIMEOUT | 500|
| - | - | socketTimeout | Socket timeout of ElasticSearch client (in milliseconds). | SW_STORAGE_ES_SOCKET_TIMEOUT | 30000|
+| - | - | numHttpClientThread | The number of threads for the underlying HTTP client to perform socket I/O.. | SW_STORAGE_ES_NUM_HTTP_CLIENT_THREAD | 2 |
Review comment:
> Is this the default of official ElasticSearch client? Should this be 2 or number of core aware?
The default of ES client is num of processors, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702520976
##########
File path: docs/en/setup/backend/configuration-vocabulary.md
##########
@@ -79,12 +79,13 @@ core|default|role|Option values: `Mixed/Receiver/Aggregator`. **Receiver** mode
| - | - | password | Nacos Auth password. | SW_CLUSTER_NACOS_PASSWORD | - |
| - | - | accessKey | Nacos Auth accessKey. | SW_CLUSTER_NACOS_ACCESSKEY | - |
| - | - | secretKey | Nacos Auth secretKey. | SW_CLUSTER_NACOS_SECRETKEY | - |
-| storage|elasticsearch| - | ElasticSearch 6 storage implementation. | - | - |
+| storage|elasticsearch| - | ElasticSearch (and OpenSearch) storage implementation. | - | - |
| - | - | nameSpace | Prefix of indexes created and used by SkyWalking. | SW_NAMESPACE | - |
| - | - | clusterNodes | ElasticSearch cluster nodes for client connection.| SW_STORAGE_ES_CLUSTER_NODES |localhost|
| - | - | protocol | HTTP or HTTPs. | SW_STORAGE_ES_HTTP_PROTOCOL | HTTP|
| - | - | connectTimeout | Connect timeout of ElasticSearch client (in milliseconds). | SW_STORAGE_ES_CONNECT_TIMEOUT | 500|
| - | - | socketTimeout | Socket timeout of ElasticSearch client (in milliseconds). | SW_STORAGE_ES_SOCKET_TIMEOUT | 30000|
+| - | - | numHttpClientThread | The number of threads for the underlying HTTP client to perform socket I/O.. | SW_STORAGE_ES_NUM_HTTP_CLIENT_THREAD | 2 |
Review comment:
Is this the default of official ElasticSearch client? Should this be 2 or number of core aware?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] JaredTan95 commented on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
JaredTan95 commented on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913330122
I tested locally, all looks great to me except `EmptyEndpointGroupException` as follows when `es` is shutdown and connected failures:
```bash
2021-09-06 11:14:10,020 org.apache.skywalking.library.elasticsearch.ElasticSearch 149 [armeria-eventloop-nio-4-2] DEBUG [] - [creqId=3fe1891d, preqId=8179d41f][http://UNKNOWN/#GET] Request: {startTime=2021-09-06T03:14:10.010Z(1630898050010987), length=0B, duration=4538ยตs(4538520ns), cause=com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException, scheme=none+http, name=GET, headers=[:method=GET, :path=/, :scheme=http, :authority=UNKNOWN]}
2021-09-06 11:14:10,021 org.apache.skywalking.library.elasticsearch.ElasticSearch 186 [armeria-eventloop-nio-4-2] WARN [] - [creqId=3fe1891d, preqId=8179d41f][http://UNKNOWN/#GET] Response: {startTime=2021-09-06T03:14:10.020Z(1630898050020718), length=0B, duration=0ns, totalDuration=9749ยตs(9749393ns), cause=com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException, headers=[:status=0]}
com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException
```
I think `EmptyEndpointGroupException` is not enough clear to point out the cause of the error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702434538
##########
File path: oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/ElasticSearch.java
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.skywalking.library.elasticsearch;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.linecorp.armeria.client.ClientFactory;
+import com.linecorp.armeria.client.WebClient;
+import com.linecorp.armeria.client.WebClientBuilder;
+import com.linecorp.armeria.client.endpoint.EndpointGroup;
+import com.linecorp.armeria.client.logging.LoggingClient;
+import com.linecorp.armeria.common.HttpData;
+import com.linecorp.armeria.common.HttpStatus;
+import com.linecorp.armeria.common.SessionProtocol;
+import com.linecorp.armeria.common.auth.BasicToken;
+import com.linecorp.armeria.common.util.Exceptions;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.util.StringUtil;
+import org.apache.skywalking.library.elasticsearch.bulk.BulkProcessorBuilder;
+import org.apache.skywalking.library.elasticsearch.client.AliasClient;
+import org.apache.skywalking.library.elasticsearch.client.DocumentClient;
+import org.apache.skywalking.library.elasticsearch.client.IndexClient;
+import org.apache.skywalking.library.elasticsearch.client.SearchClient;
+import org.apache.skywalking.library.elasticsearch.client.TemplateClient;
+import org.apache.skywalking.library.elasticsearch.requests.search.Search;
+import org.apache.skywalking.library.elasticsearch.response.NodeInfo;
+import org.apache.skywalking.library.elasticsearch.response.search.SearchResponse;
+
+@Slf4j
+public final class ElasticSearch implements Closeable {
+ private final ObjectMapper mapper = new ObjectMapper()
+ .setSerializationInclusion(JsonInclude.Include.NON_NULL)
+ .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+
+ private final WebClient client;
+
+ private final ClientFactory clientFactory;
+
+ private final CompletableFuture<ElasticSearchVersion> version;
+
+ private final TemplateClient templateClient;
+
+ private final IndexClient indexClient;
+
+ private final DocumentClient documentClient;
+
+ private final AliasClient aliasClient;
+
+ private final SearchClient searchClient;
+
+ ElasticSearch(SessionProtocol protocol,
+ String username, String password,
+ EndpointGroup endpointGroup,
+ ClientFactory clientFactory) {
+ this.clientFactory = clientFactory;
+ this.version = new CompletableFuture<>();
+
+ final WebClientBuilder builder =
+ WebClient.builder(protocol, endpointGroup)
+ .factory(clientFactory)
+ .decorator(
+ LoggingClient.builder()
+ .logger(log)
+ .newDecorator());
Review comment:
How should we set up thread number of the web client(or netty)? I think this matters in query or metrics(including metadata) update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702422057
##########
File path: docs/en/setup/backend/backend-storage.md
##########
@@ -34,23 +34,18 @@ storage:
## OpenSearch
-OpenSearch storage shares the same configurations as ElasticSearch 7.
-In order to activate ElasticSearch 7 as storage, set storage provider to **elasticsearch7**.
-Please download the `apache-skywalking-bin-es7.tar.gz` if you want to use OpenSearch as storage.
+OpenSearch storage shares the same configurations as ElasticSearch.
+In order to activate OpenSearch as storage, set storage provider to **elasticsearch**.
## ElasticSearch
**NOTE:** Elastic announced through their blog that Elasticsearch will be moving over to a Server Side Public
License (SSPL), which is incompatible with Apache License 2.0. This license change is effective from Elasticsearch
version 7.11. So please choose the suitable ElasticSearch version according to your usage.
-- In order to activate ElasticSearch 6 as storage, set storage provider to **elasticsearch**
-- In order to activate ElasticSearch 7 as storage, set storage provider to **elasticsearch7**
-
-**Required ElasticSearch 6.3.2 or higher. HTTP RestHighLevelClient is used to connect server.**
-
-- For ElasticSearch 6.3.2 ~ 7.0.0 (excluded), please download `apache-skywalking-bin.tar.gz`.
-- For ElasticSearch 7.0.0 ~ 8.0.0 (excluded), please download `apache-skywalking-bin-es7.tar.gz`.
+Since 8.8.0, SkyWalking rebuilds the ElasticSearch client on top of ElasticSearch REST API and automatically picks up
+correct request formats according to the server side version, hence you don't need to download different binaries
+and don't need to configure different storage selector for different ElasticSearch server side version anymore.
Review comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913378200
Thank you @wu-sheng @JaredTan95 for reviewing and checking, I'm going to merge this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913378740
@wankai123 is confirming too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 removed a comment on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 removed a comment on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913381194
> `docker/oap/Dockerfile.oap` will be fixed in follow-up PRs?
~~I think all needed changes to docker (and `docker/oap/Dockerfile.oap`) are included in this PR, the official docker images will be done in http://github.com/apache/skywalking-docker after this PR, if I missed anything, please let me know @hanahmily~~
ignore this too ๐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r700339532
##########
File path: oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/ElasticSearchClientBuilder.java
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.skywalking.library.elasticsearch;
+
+import com.google.common.collect.ImmutableList;
+import com.linecorp.armeria.client.ClientFactory;
+import com.linecorp.armeria.client.ClientFactoryBuilder;
+import com.linecorp.armeria.client.Endpoint;
+import com.linecorp.armeria.client.endpoint.EndpointGroup;
+import com.linecorp.armeria.client.endpoint.healthcheck.HealthCheckedEndpointGroup;
+import com.linecorp.armeria.common.SessionProtocol;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.KeyStore;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+import javax.net.ssl.TrustManagerFactory;
+import lombok.SneakyThrows;
+import org.apache.skywalking.apm.util.StringUtil;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static java.util.Objects.requireNonNull;
+import static org.apache.skywalking.apm.util.StringUtil.isNotBlank;
+
+public final class ElasticSearchClientBuilder {
+ private SessionProtocol protocol = SessionProtocol.HTTP;
+
+ private String username;
+
+ private String password;
+
+ private Duration healthCheckRetryInterval = Duration.ofSeconds(30);
+
+ private final ImmutableList.Builder<String> endpoints = ImmutableList.builder();
+
+ private String trustStorePath;
+
+ private String trustStorePass;
+
+ private Duration connectTimeout = Duration.ofMillis(500);
+
+ public ElasticSearchClientBuilder protocol(String protocol) {
+ checkArgument(isNotBlank(protocol), "protocol cannot be blank");
+ this.protocol = SessionProtocol.of(protocol);
+ return this;
+ }
+
+ public ElasticSearchClientBuilder username(String username) {
+ this.username = requireNonNull(username, "username");
+ return this;
+ }
+
+ public ElasticSearchClientBuilder password(String password) {
+ this.password = requireNonNull(password, "password");
+ return this;
+ }
+
+ public ElasticSearchClientBuilder endpoints(Iterable<String> endpoints) {
+ requireNonNull(endpoints, "endpoints");
+ this.endpoints.addAll(endpoints);
+ return this;
+ }
+
+ public ElasticSearchClientBuilder endpoints(String... endpoints) {
+ return endpoints(Arrays.asList(endpoints));
+ }
+
+ public ElasticSearchClientBuilder healthCheckRetryInterval(Duration healthCheckRetryInterval) {
+ requireNonNull(healthCheckRetryInterval, "healthCheckRetryInterval");
+ this.healthCheckRetryInterval = healthCheckRetryInterval;
+ return this;
+ }
+
+ public ElasticSearchClientBuilder trustStorePath(String trustStorePath) {
+ requireNonNull(trustStorePath, "trustStorePath");
+ this.trustStorePath = trustStorePath;
+ return this;
+ }
+
+ public ElasticSearchClientBuilder trustStorePass(String trustStorePass) {
+ requireNonNull(trustStorePass, "trustStorePass");
+ this.trustStorePass = trustStorePass;
+ return this;
+ }
+
+ public ElasticSearchClientBuilder connectTimeout(int connectTimeout) {
+ checkArgument(connectTimeout > 0, "connectTimeout must be positive");
+ this.connectTimeout = Duration.ofMillis(connectTimeout);
+ return this;
+ }
+
+ @SneakyThrows
+ public ElasticSearchClient build() {
+ final List<Endpoint> endpoints =
+ this.endpoints.build().stream().filter(StringUtil::isNotBlank).map(it -> {
+ final String[] parts = it.split(":", 2);
+ if (parts.length == 2) {
+ return Endpoint.of(parts[0], Integer.parseInt(parts[1]));
+ }
+ return Endpoint.of(parts[0]);
+ }).collect(Collectors.toList());
+ final HealthCheckedEndpointGroup endpointGroup =
+ HealthCheckedEndpointGroup.builder(EndpointGroup.of(endpoints), "_cluster/health")
+ .protocol(protocol)
+ .useGet(true)
+ .retryInterval(healthCheckRetryInterval)
+ .build();
+ final ClientFactoryBuilder factoryBuilder = ClientFactory.builder();
+ factoryBuilder.connectTimeout(connectTimeout);
+
+ if (StringUtil.isNotBlank(trustStorePath)) {
+ final TrustManagerFactory trustManagerFactory =
+ TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
+ final KeyStore truststore = KeyStore.getInstance("jks");
+ try (final InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
Review comment:
*PATH_TRAVERSAL_IN:* This API (java/nio/file/Paths.get(Ljava/lang/String;[Ljava/lang/String;)Ljava/nio/file/Path;) reads a file whose location might be specified by user input [(details)](https://find-sec-bugs.github.io/bugs.htm#PATH_TRAVERSAL_IN)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
}
- public SearchResponse search(String indexName, SearchSourceBuilder searchSourceBuilder) throws IOException {
+ public SearchResponse search(String indexName, Search search) {
indexName = formatIndexName(indexName);
- return doSearch(searchSourceBuilder, indexName);
+ return esClient.search(search, indexName);
+ }
+
+ protected SearchResponse doSearch(Search search, String... indexNames) {
+ return esClient.search(
+ search,
+ ImmutableMap.of(
+ "ignore_unavailable", true,
+ "allow_no_indices", true,
+ "expand_wildcards", "open"
+ ),
+ indexNames
+ );
}
- protected SearchResponse doSearch(SearchSourceBuilder searchSourceBuilder,
- String... indexNames) throws IOException {
- SearchRequest searchRequest = new SearchRequest(indexNames);
- searchRequest.indicesOptions(IndicesOptions.fromOptions(true, true, true, false));
- searchRequest.types(TYPE);
- searchRequest.source(searchSourceBuilder);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- handleIOPoolStopped(t);
- throw t;
- }
- }
+ public Optional<Document> get(String indexName, String id) {
+ indexName = formatIndexName(indexName);
- protected void handleIOPoolStopped(Throwable t) throws IOException {
- if (!(t instanceof IllegalStateException)) {
- return;
- }
- IllegalStateException ise = (IllegalStateException) t;
- // Fixed the issue described in https://github.com/elastic/elasticsearch/issues/39946
- if (ise.getMessage().contains("I/O reactor status: STOPPED") &&
- connectLock.tryLock()) {
- try {
- connect();
- } catch (KeyStoreException | NoSuchAlgorithmException | KeyManagementException | CertificateException e) {
- throw new IllegalStateException("Can't reconnect to Elasticsearch", e);
- }
- }
+ return esClient.documents().get(indexName, TYPE, id);
}
- public GetResponse get(String indexName, String id) throws IOException {
+ public boolean existDoc(String indexName, String id) {
indexName = formatIndexName(indexName);
- GetRequest request = new GetRequest(indexName, TYPE, id);
- try {
- GetResponse response = client.get(request);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+
+ return esClient.documents().exists(indexName, TYPE, id);
}
- public SearchResponse ids(String indexName, String[] ids) throws IOException {
+ public Optional<Documents> ids(String indexName, Iterable<String> ids) {
indexName = formatIndexName(indexName);
- SearchRequest searchRequest = new SearchRequest(indexName);
- searchRequest.types(TYPE);
- searchRequest.source().query(QueryBuilders.idsQuery().addIds(ids)).size(ids.length);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+ return esClient.documents().mget(indexName, TYPE, ids);
}
- public void forceInsert(String indexName, String id, XContentBuilder source) throws IOException {
- IndexRequest request = (IndexRequest) prepareInsert(indexName, id, source);
- request.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
+ public Optional<Documents> ids(String indexName, String[] ids) {
+ return ids(indexName, Arrays.asList(ids));
+ }
+
+ public void forceInsert(String indexName, String id, Map<String, Object> source) {
+ IndexRequestWrapper wrapper = prepareInsert(indexName, id, source);
+ Map<String, Object> params = ImmutableMap.of("refresh", "true");
try {
- client.index(request);
+ esClient.documents().index(wrapper.getRequest(), params);
healthChecker.health();
} catch (Throwable t) {
healthChecker.unHealth(t);
throw t;
}
}
- public void forceUpdate(String indexName, String id, XContentBuilder source) throws IOException {
- org.elasticsearch.action.update.UpdateRequest request = (org.elasticsearch.action.update.UpdateRequest) prepareUpdate(
- indexName, id, source);
- request.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
+ public void forceUpdate(String indexName, String id, Map<String, Object> source) {
+ UpdateRequestWrapper wrapper = prepareUpdate(indexName, id, source);
+ Map<String, Object> params = ImmutableMap.of("refresh", "true");
try {
- client.update(request);
+ esClient.documents().update(wrapper.getRequest(), params);
healthChecker.health();
} catch (Throwable t) {
healthChecker.unHealth(t);
throw t;
}
}
- public InsertRequest prepareInsert(String indexName, String id, XContentBuilder source) {
+ public IndexRequestWrapper prepareInsert(String indexName, String id,
+ Map<String, Object> source) {
indexName = formatIndexName(indexName);
- return new ElasticSearchInsertRequest(indexName, TYPE, id).source(source);
+ return new IndexRequestWrapper(indexName, TYPE, id, source);
}
- public UpdateRequest prepareUpdate(String indexName, String id, XContentBuilder source) {
+ public UpdateRequestWrapper prepareUpdate(String indexName, String id,
+ Map<String, Object> source) {
indexName = formatIndexName(indexName);
- return new ElasticSearchUpdateRequest(indexName, TYPE, id).doc(source);
+ return new UpdateRequestWrapper(indexName, TYPE, id, source);
}
- public int delete(String indexName, String timeBucketColumnName, long endTimeBucket) throws IOException {
+ public void delete(String indexName, String timeBucketColumnName, long endTimeBucket) {
indexName = formatIndexName(indexName);
- Map<String, String> params = Collections.singletonMap("conflicts", "proceed");
- String jsonString = "{" + " \"query\": {" + " \"range\": {" + " \"" + timeBucketColumnName + "\": {" + " \"lte\": " + endTimeBucket + " }" + " }" + " }" + "}";
- HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPost.METHOD_NAME, "/" + indexName + "/_delete_by_query", params, entity);
- log.debug("delete indexName: {}, jsonString : {}", indexName, jsonString);
- return response.getStatusLine().getStatusCode();
- }
+ final Map<String, Object> params = Collections.singletonMap("conflicts", "proceed");
+ final Query query = Query.range(timeBucketColumnName).lte(endTimeBucket).build();
- /**
- * @since 8.7.0 SkyWalking don't use sync bulk anymore. This method is just kept for unexpected case in the future.
- */
- @Deprecated
- public void synchronousBulk(BulkRequest request) {
- request.timeout(TimeValue.timeValueMinutes(2));
- request.setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);
- request.waitForActiveShards(ActiveShardCount.ONE);
- try {
- int size = request.requests().size();
- BulkResponse responses = client.bulk(request);
- log.info("Synchronous bulk took time: {} millis, size: {}", responses.getTook().getMillis(), size);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- }
- }
-
- public BulkProcessor createBulkProcessor(int bulkActions, int flushInterval, int concurrentRequests) {
- BulkProcessor.Listener listener = createBulkListener();
-
- return BulkProcessor.builder(client::bulkAsync, listener)
- .setBulkActions(bulkActions)
- .setFlushInterval(TimeValue.timeValueSeconds(flushInterval))
- .setConcurrentRequests(concurrentRequests)
- .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(100), 3))
- .build();
+ esClient.documents().delete(indexName, TYPE, query, params);
}
- protected BulkProcessor.Listener createBulkListener() {
- return new BulkProcessor.Listener() {
- @Override
- public void beforeBulk(long executionId, BulkRequest request) {
- int numberOfActions = request.numberOfActions();
- log.debug("Executing bulk [{}] with {} requests", executionId, numberOfActions);
- }
-
- @Override
- public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
- if (response.hasFailures()) {
- log.warn("Bulk [{}] executed with failures:[{}]", executionId, response.buildFailureMessage());
- } else {
- log.info(
- "Bulk execution id [{}] completed in {} milliseconds, size: {}", executionId, response.getTook()
- .getMillis(),
- request
- .requests()
- .size()
- );
- }
- }
-
- @Override
- public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
- log.error("Failed to execute bulk", failure);
- }
- };
+ public BulkProcessor createBulkProcessor(int bulkActions,
+ int flushInterval,
+ int concurrentRequests) {
+ return esClient.bulkProcessor()
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.createBulkProcessor(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.createIndex(...)` indirectly reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.createIndex(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.createOrUpdateTemplate(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
}
- public SearchResponse search(String indexName, SearchSourceBuilder searchSourceBuilder) throws IOException {
+ public SearchResponse search(String indexName, Search search) {
indexName = formatIndexName(indexName);
- return doSearch(searchSourceBuilder, indexName);
+ return esClient.search(search, indexName);
+ }
+
+ protected SearchResponse doSearch(Search search, String... indexNames) {
+ return esClient.search(
+ search,
+ ImmutableMap.of(
+ "ignore_unavailable", true,
+ "allow_no_indices", true,
+ "expand_wildcards", "open"
+ ),
+ indexNames
+ );
}
- protected SearchResponse doSearch(SearchSourceBuilder searchSourceBuilder,
- String... indexNames) throws IOException {
- SearchRequest searchRequest = new SearchRequest(indexNames);
- searchRequest.indicesOptions(IndicesOptions.fromOptions(true, true, true, false));
- searchRequest.types(TYPE);
- searchRequest.source(searchSourceBuilder);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- handleIOPoolStopped(t);
- throw t;
- }
- }
+ public Optional<Document> get(String indexName, String id) {
+ indexName = formatIndexName(indexName);
- protected void handleIOPoolStopped(Throwable t) throws IOException {
- if (!(t instanceof IllegalStateException)) {
- return;
- }
- IllegalStateException ise = (IllegalStateException) t;
- // Fixed the issue described in https://github.com/elastic/elasticsearch/issues/39946
- if (ise.getMessage().contains("I/O reactor status: STOPPED") &&
- connectLock.tryLock()) {
- try {
- connect();
- } catch (KeyStoreException | NoSuchAlgorithmException | KeyManagementException | CertificateException e) {
- throw new IllegalStateException("Can't reconnect to Elasticsearch", e);
- }
- }
+ return esClient.documents().get(indexName, TYPE, id);
}
- public GetResponse get(String indexName, String id) throws IOException {
+ public boolean existDoc(String indexName, String id) {
indexName = formatIndexName(indexName);
- GetRequest request = new GetRequest(indexName, TYPE, id);
- try {
- GetResponse response = client.get(request);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+
+ return esClient.documents().exists(indexName, TYPE, id);
}
- public SearchResponse ids(String indexName, String[] ids) throws IOException {
+ public Optional<Documents> ids(String indexName, Iterable<String> ids) {
indexName = formatIndexName(indexName);
- SearchRequest searchRequest = new SearchRequest(indexName);
- searchRequest.types(TYPE);
- searchRequest.source().query(QueryBuilders.idsQuery().addIds(ids)).size(ids.length);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+ return esClient.documents().mget(indexName, TYPE, ids);
}
- public void forceInsert(String indexName, String id, XContentBuilder source) throws IOException {
- IndexRequest request = (IndexRequest) prepareInsert(indexName, id, source);
- request.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
+ public Optional<Documents> ids(String indexName, String[] ids) {
+ return ids(indexName, Arrays.asList(ids));
+ }
+
+ public void forceInsert(String indexName, String id, Map<String, Object> source) {
+ IndexRequestWrapper wrapper = prepareInsert(indexName, id, source);
+ Map<String, Object> params = ImmutableMap.of("refresh", "true");
try {
- client.index(request);
+ esClient.documents().index(wrapper.getRequest(), params);
healthChecker.health();
} catch (Throwable t) {
healthChecker.unHealth(t);
throw t;
}
}
- public void forceUpdate(String indexName, String id, XContentBuilder source) throws IOException {
- org.elasticsearch.action.update.UpdateRequest request = (org.elasticsearch.action.update.UpdateRequest) prepareUpdate(
- indexName, id, source);
- request.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
+ public void forceUpdate(String indexName, String id, Map<String, Object> source) {
+ UpdateRequestWrapper wrapper = prepareUpdate(indexName, id, source);
+ Map<String, Object> params = ImmutableMap.of("refresh", "true");
try {
- client.update(request);
+ esClient.documents().update(wrapper.getRequest(), params);
healthChecker.health();
} catch (Throwable t) {
healthChecker.unHealth(t);
throw t;
}
}
- public InsertRequest prepareInsert(String indexName, String id, XContentBuilder source) {
+ public IndexRequestWrapper prepareInsert(String indexName, String id,
+ Map<String, Object> source) {
indexName = formatIndexName(indexName);
- return new ElasticSearchInsertRequest(indexName, TYPE, id).source(source);
+ return new IndexRequestWrapper(indexName, TYPE, id, source);
}
- public UpdateRequest prepareUpdate(String indexName, String id, XContentBuilder source) {
+ public UpdateRequestWrapper prepareUpdate(String indexName, String id,
+ Map<String, Object> source) {
indexName = formatIndexName(indexName);
- return new ElasticSearchUpdateRequest(indexName, TYPE, id).doc(source);
+ return new UpdateRequestWrapper(indexName, TYPE, id, source);
}
- public int delete(String indexName, String timeBucketColumnName, long endTimeBucket) throws IOException {
+ public void delete(String indexName, String timeBucketColumnName, long endTimeBucket) {
indexName = formatIndexName(indexName);
- Map<String, String> params = Collections.singletonMap("conflicts", "proceed");
- String jsonString = "{" + " \"query\": {" + " \"range\": {" + " \"" + timeBucketColumnName + "\": {" + " \"lte\": " + endTimeBucket + " }" + " }" + " }" + "}";
- HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPost.METHOD_NAME, "/" + indexName + "/_delete_by_query", params, entity);
- log.debug("delete indexName: {}, jsonString : {}", indexName, jsonString);
- return response.getStatusLine().getStatusCode();
- }
+ final Map<String, Object> params = Collections.singletonMap("conflicts", "proceed");
+ final Query query = Query.range(timeBucketColumnName).lte(endTimeBucket).build();
- /**
- * @since 8.7.0 SkyWalking don't use sync bulk anymore. This method is just kept for unexpected case in the future.
- */
- @Deprecated
- public void synchronousBulk(BulkRequest request) {
- request.timeout(TimeValue.timeValueMinutes(2));
- request.setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);
- request.waitForActiveShards(ActiveShardCount.ONE);
- try {
- int size = request.requests().size();
- BulkResponse responses = client.bulk(request);
- log.info("Synchronous bulk took time: {} millis, size: {}", responses.getTook().getMillis(), size);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- }
- }
-
- public BulkProcessor createBulkProcessor(int bulkActions, int flushInterval, int concurrentRequests) {
- BulkProcessor.Listener listener = createBulkListener();
-
- return BulkProcessor.builder(client::bulkAsync, listener)
- .setBulkActions(bulkActions)
- .setFlushInterval(TimeValue.timeValueSeconds(flushInterval))
- .setConcurrentRequests(concurrentRequests)
- .setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(100), 3))
- .build();
+ esClient.documents().delete(indexName, TYPE, query, params);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.delete(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.deleteByIndexName(...)` indirectly reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.deleteByModelName(...)` indirectly reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.deleteIndex(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.deleteTemplate(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
}
- public SearchResponse search(String indexName, SearchSourceBuilder searchSourceBuilder) throws IOException {
+ public SearchResponse search(String indexName, Search search) {
indexName = formatIndexName(indexName);
- return doSearch(searchSourceBuilder, indexName);
+ return esClient.search(search, indexName);
+ }
+
+ protected SearchResponse doSearch(Search search, String... indexNames) {
+ return esClient.search(
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.doSearch(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
}
- public SearchResponse search(String indexName, SearchSourceBuilder searchSourceBuilder) throws IOException {
+ public SearchResponse search(String indexName, Search search) {
indexName = formatIndexName(indexName);
- return doSearch(searchSourceBuilder, indexName);
+ return esClient.search(search, indexName);
+ }
+
+ protected SearchResponse doSearch(Search search, String... indexNames) {
+ return esClient.search(
+ search,
+ ImmutableMap.of(
+ "ignore_unavailable", true,
+ "allow_no_indices", true,
+ "expand_wildcards", "open"
+ ),
+ indexNames
+ );
}
- protected SearchResponse doSearch(SearchSourceBuilder searchSourceBuilder,
- String... indexNames) throws IOException {
- SearchRequest searchRequest = new SearchRequest(indexNames);
- searchRequest.indicesOptions(IndicesOptions.fromOptions(true, true, true, false));
- searchRequest.types(TYPE);
- searchRequest.source(searchSourceBuilder);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- handleIOPoolStopped(t);
- throw t;
- }
- }
+ public Optional<Document> get(String indexName, String id) {
+ indexName = formatIndexName(indexName);
- protected void handleIOPoolStopped(Throwable t) throws IOException {
- if (!(t instanceof IllegalStateException)) {
- return;
- }
- IllegalStateException ise = (IllegalStateException) t;
- // Fixed the issue described in https://github.com/elastic/elasticsearch/issues/39946
- if (ise.getMessage().contains("I/O reactor status: STOPPED") &&
- connectLock.tryLock()) {
- try {
- connect();
- } catch (KeyStoreException | NoSuchAlgorithmException | KeyManagementException | CertificateException e) {
- throw new IllegalStateException("Can't reconnect to Elasticsearch", e);
- }
- }
+ return esClient.documents().get(indexName, TYPE, id);
}
- public GetResponse get(String indexName, String id) throws IOException {
+ public boolean existDoc(String indexName, String id) {
indexName = formatIndexName(indexName);
- GetRequest request = new GetRequest(indexName, TYPE, id);
- try {
- GetResponse response = client.get(request);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+
+ return esClient.documents().exists(indexName, TYPE, id);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.existDoc(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
}
- public SearchResponse search(String indexName, SearchSourceBuilder searchSourceBuilder) throws IOException {
+ public SearchResponse search(String indexName, Search search) {
indexName = formatIndexName(indexName);
- return doSearch(searchSourceBuilder, indexName);
+ return esClient.search(search, indexName);
+ }
+
+ protected SearchResponse doSearch(Search search, String... indexNames) {
+ return esClient.search(
+ search,
+ ImmutableMap.of(
+ "ignore_unavailable", true,
+ "allow_no_indices", true,
+ "expand_wildcards", "open"
+ ),
+ indexNames
+ );
}
- protected SearchResponse doSearch(SearchSourceBuilder searchSourceBuilder,
- String... indexNames) throws IOException {
- SearchRequest searchRequest = new SearchRequest(indexNames);
- searchRequest.indicesOptions(IndicesOptions.fromOptions(true, true, true, false));
- searchRequest.types(TYPE);
- searchRequest.source(searchSourceBuilder);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- handleIOPoolStopped(t);
- throw t;
- }
- }
+ public Optional<Document> get(String indexName, String id) {
+ indexName = formatIndexName(indexName);
- protected void handleIOPoolStopped(Throwable t) throws IOException {
- if (!(t instanceof IllegalStateException)) {
- return;
- }
- IllegalStateException ise = (IllegalStateException) t;
- // Fixed the issue described in https://github.com/elastic/elasticsearch/issues/39946
- if (ise.getMessage().contains("I/O reactor status: STOPPED") &&
- connectLock.tryLock()) {
- try {
- connect();
- } catch (KeyStoreException | NoSuchAlgorithmException | KeyManagementException | CertificateException e) {
- throw new IllegalStateException("Can't reconnect to Elasticsearch", e);
- }
- }
+ return esClient.documents().get(indexName, TYPE, id);
}
- public GetResponse get(String indexName, String id) throws IOException {
+ public boolean existDoc(String indexName, String id) {
indexName = formatIndexName(indexName);
- GetRequest request = new GetRequest(indexName, TYPE, id);
- try {
- GetResponse response = client.get(request);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+
+ return esClient.documents().exists(indexName, TYPE, id);
}
- public SearchResponse ids(String indexName, String[] ids) throws IOException {
+ public Optional<Documents> ids(String indexName, Iterable<String> ids) {
indexName = formatIndexName(indexName);
- SearchRequest searchRequest = new SearchRequest(indexName);
- searchRequest.types(TYPE);
- searchRequest.source().query(QueryBuilders.idsQuery().addIds(ids)).size(ids.length);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+ return esClient.documents().mget(indexName, TYPE, ids);
}
- public void forceInsert(String indexName, String id, XContentBuilder source) throws IOException {
- IndexRequest request = (IndexRequest) prepareInsert(indexName, id, source);
- request.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
+ public Optional<Documents> ids(String indexName, String[] ids) {
+ return ids(indexName, Arrays.asList(ids));
+ }
+
+ public void forceInsert(String indexName, String id, Map<String, Object> source) {
+ IndexRequestWrapper wrapper = prepareInsert(indexName, id, source);
+ Map<String, Object> params = ImmutableMap.of("refresh", "true");
try {
- client.index(request);
+ esClient.documents().index(wrapper.getRequest(), params);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.forceInsert(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
}
- public SearchResponse search(String indexName, SearchSourceBuilder searchSourceBuilder) throws IOException {
+ public SearchResponse search(String indexName, Search search) {
indexName = formatIndexName(indexName);
- return doSearch(searchSourceBuilder, indexName);
+ return esClient.search(search, indexName);
+ }
+
+ protected SearchResponse doSearch(Search search, String... indexNames) {
+ return esClient.search(
+ search,
+ ImmutableMap.of(
+ "ignore_unavailable", true,
+ "allow_no_indices", true,
+ "expand_wildcards", "open"
+ ),
+ indexNames
+ );
}
- protected SearchResponse doSearch(SearchSourceBuilder searchSourceBuilder,
- String... indexNames) throws IOException {
- SearchRequest searchRequest = new SearchRequest(indexNames);
- searchRequest.indicesOptions(IndicesOptions.fromOptions(true, true, true, false));
- searchRequest.types(TYPE);
- searchRequest.source(searchSourceBuilder);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- handleIOPoolStopped(t);
- throw t;
- }
- }
+ public Optional<Document> get(String indexName, String id) {
+ indexName = formatIndexName(indexName);
- protected void handleIOPoolStopped(Throwable t) throws IOException {
- if (!(t instanceof IllegalStateException)) {
- return;
- }
- IllegalStateException ise = (IllegalStateException) t;
- // Fixed the issue described in https://github.com/elastic/elasticsearch/issues/39946
- if (ise.getMessage().contains("I/O reactor status: STOPPED") &&
- connectLock.tryLock()) {
- try {
- connect();
- } catch (KeyStoreException | NoSuchAlgorithmException | KeyManagementException | CertificateException e) {
- throw new IllegalStateException("Can't reconnect to Elasticsearch", e);
- }
- }
+ return esClient.documents().get(indexName, TYPE, id);
}
- public GetResponse get(String indexName, String id) throws IOException {
+ public boolean existDoc(String indexName, String id) {
indexName = formatIndexName(indexName);
- GetRequest request = new GetRequest(indexName, TYPE, id);
- try {
- GetResponse response = client.get(request);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+
+ return esClient.documents().exists(indexName, TYPE, id);
}
- public SearchResponse ids(String indexName, String[] ids) throws IOException {
+ public Optional<Documents> ids(String indexName, Iterable<String> ids) {
indexName = formatIndexName(indexName);
- SearchRequest searchRequest = new SearchRequest(indexName);
- searchRequest.types(TYPE);
- searchRequest.source().query(QueryBuilders.idsQuery().addIds(ids)).size(ids.length);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+ return esClient.documents().mget(indexName, TYPE, ids);
}
- public void forceInsert(String indexName, String id, XContentBuilder source) throws IOException {
- IndexRequest request = (IndexRequest) prepareInsert(indexName, id, source);
- request.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
+ public Optional<Documents> ids(String indexName, String[] ids) {
+ return ids(indexName, Arrays.asList(ids));
+ }
+
+ public void forceInsert(String indexName, String id, Map<String, Object> source) {
+ IndexRequestWrapper wrapper = prepareInsert(indexName, id, source);
+ Map<String, Object> params = ImmutableMap.of("refresh", "true");
try {
- client.index(request);
+ esClient.documents().index(wrapper.getRequest(), params);
healthChecker.health();
} catch (Throwable t) {
healthChecker.unHealth(t);
throw t;
}
}
- public void forceUpdate(String indexName, String id, XContentBuilder source) throws IOException {
- org.elasticsearch.action.update.UpdateRequest request = (org.elasticsearch.action.update.UpdateRequest) prepareUpdate(
- indexName, id, source);
- request.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
+ public void forceUpdate(String indexName, String id, Map<String, Object> source) {
+ UpdateRequestWrapper wrapper = prepareUpdate(indexName, id, source);
+ Map<String, Object> params = ImmutableMap.of("refresh", "true");
try {
- client.update(request);
+ esClient.documents().update(wrapper.getRequest(), params);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.forceUpdate(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
}
- public SearchResponse search(String indexName, SearchSourceBuilder searchSourceBuilder) throws IOException {
+ public SearchResponse search(String indexName, Search search) {
indexName = formatIndexName(indexName);
- return doSearch(searchSourceBuilder, indexName);
+ return esClient.search(search, indexName);
+ }
+
+ protected SearchResponse doSearch(Search search, String... indexNames) {
+ return esClient.search(
+ search,
+ ImmutableMap.of(
+ "ignore_unavailable", true,
+ "allow_no_indices", true,
+ "expand_wildcards", "open"
+ ),
+ indexNames
+ );
}
- protected SearchResponse doSearch(SearchSourceBuilder searchSourceBuilder,
- String... indexNames) throws IOException {
- SearchRequest searchRequest = new SearchRequest(indexNames);
- searchRequest.indicesOptions(IndicesOptions.fromOptions(true, true, true, false));
- searchRequest.types(TYPE);
- searchRequest.source(searchSourceBuilder);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- handleIOPoolStopped(t);
- throw t;
- }
- }
+ public Optional<Document> get(String indexName, String id) {
+ indexName = formatIndexName(indexName);
- protected void handleIOPoolStopped(Throwable t) throws IOException {
- if (!(t instanceof IllegalStateException)) {
- return;
- }
- IllegalStateException ise = (IllegalStateException) t;
- // Fixed the issue described in https://github.com/elastic/elasticsearch/issues/39946
- if (ise.getMessage().contains("I/O reactor status: STOPPED") &&
- connectLock.tryLock()) {
- try {
- connect();
- } catch (KeyStoreException | NoSuchAlgorithmException | KeyManagementException | CertificateException e) {
- throw new IllegalStateException("Can't reconnect to Elasticsearch", e);
- }
- }
+ return esClient.documents().get(indexName, TYPE, id);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.get(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.getIndex(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.getTemplate(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
}
- public SearchResponse search(String indexName, SearchSourceBuilder searchSourceBuilder) throws IOException {
+ public SearchResponse search(String indexName, Search search) {
indexName = formatIndexName(indexName);
- return doSearch(searchSourceBuilder, indexName);
+ return esClient.search(search, indexName);
+ }
+
+ protected SearchResponse doSearch(Search search, String... indexNames) {
+ return esClient.search(
+ search,
+ ImmutableMap.of(
+ "ignore_unavailable", true,
+ "allow_no_indices", true,
+ "expand_wildcards", "open"
+ ),
+ indexNames
+ );
}
- protected SearchResponse doSearch(SearchSourceBuilder searchSourceBuilder,
- String... indexNames) throws IOException {
- SearchRequest searchRequest = new SearchRequest(indexNames);
- searchRequest.indicesOptions(IndicesOptions.fromOptions(true, true, true, false));
- searchRequest.types(TYPE);
- searchRequest.source(searchSourceBuilder);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- handleIOPoolStopped(t);
- throw t;
- }
- }
+ public Optional<Document> get(String indexName, String id) {
+ indexName = formatIndexName(indexName);
- protected void handleIOPoolStopped(Throwable t) throws IOException {
- if (!(t instanceof IllegalStateException)) {
- return;
- }
- IllegalStateException ise = (IllegalStateException) t;
- // Fixed the issue described in https://github.com/elastic/elasticsearch/issues/39946
- if (ise.getMessage().contains("I/O reactor status: STOPPED") &&
- connectLock.tryLock()) {
- try {
- connect();
- } catch (KeyStoreException | NoSuchAlgorithmException | KeyManagementException | CertificateException e) {
- throw new IllegalStateException("Can't reconnect to Elasticsearch", e);
- }
- }
+ return esClient.documents().get(indexName, TYPE, id);
}
- public GetResponse get(String indexName, String id) throws IOException {
+ public boolean existDoc(String indexName, String id) {
indexName = formatIndexName(indexName);
- GetRequest request = new GetRequest(indexName, TYPE, id);
- try {
- GetResponse response = client.get(request);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+
+ return esClient.documents().exists(indexName, TYPE, id);
}
- public SearchResponse ids(String indexName, String[] ids) throws IOException {
+ public Optional<Documents> ids(String indexName, Iterable<String> ids) {
indexName = formatIndexName(indexName);
- SearchRequest searchRequest = new SearchRequest(indexName);
- searchRequest.types(TYPE);
- searchRequest.source().query(QueryBuilders.idsQuery().addIds(ids)).size(ids.length);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+ return esClient.documents().mget(indexName, TYPE, ids);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.ids(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
}
- public SearchResponse search(String indexName, SearchSourceBuilder searchSourceBuilder) throws IOException {
+ public SearchResponse search(String indexName, Search search) {
indexName = formatIndexName(indexName);
- return doSearch(searchSourceBuilder, indexName);
+ return esClient.search(search, indexName);
+ }
+
+ protected SearchResponse doSearch(Search search, String... indexNames) {
+ return esClient.search(
+ search,
+ ImmutableMap.of(
+ "ignore_unavailable", true,
+ "allow_no_indices", true,
+ "expand_wildcards", "open"
+ ),
+ indexNames
+ );
}
- protected SearchResponse doSearch(SearchSourceBuilder searchSourceBuilder,
- String... indexNames) throws IOException {
- SearchRequest searchRequest = new SearchRequest(indexNames);
- searchRequest.indicesOptions(IndicesOptions.fromOptions(true, true, true, false));
- searchRequest.types(TYPE);
- searchRequest.source(searchSourceBuilder);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- handleIOPoolStopped(t);
- throw t;
- }
- }
+ public Optional<Document> get(String indexName, String id) {
+ indexName = formatIndexName(indexName);
- protected void handleIOPoolStopped(Throwable t) throws IOException {
- if (!(t instanceof IllegalStateException)) {
- return;
- }
- IllegalStateException ise = (IllegalStateException) t;
- // Fixed the issue described in https://github.com/elastic/elasticsearch/issues/39946
- if (ise.getMessage().contains("I/O reactor status: STOPPED") &&
- connectLock.tryLock()) {
- try {
- connect();
- } catch (KeyStoreException | NoSuchAlgorithmException | KeyManagementException | CertificateException e) {
- throw new IllegalStateException("Can't reconnect to Elasticsearch", e);
- }
- }
+ return esClient.documents().get(indexName, TYPE, id);
}
- public GetResponse get(String indexName, String id) throws IOException {
+ public boolean existDoc(String indexName, String id) {
indexName = formatIndexName(indexName);
- GetRequest request = new GetRequest(indexName, TYPE, id);
- try {
- GetResponse response = client.get(request);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+
+ return esClient.documents().exists(indexName, TYPE, id);
}
- public SearchResponse ids(String indexName, String[] ids) throws IOException {
+ public Optional<Documents> ids(String indexName, Iterable<String> ids) {
indexName = formatIndexName(indexName);
- SearchRequest searchRequest = new SearchRequest(indexName);
- searchRequest.types(TYPE);
- searchRequest.source().query(QueryBuilders.idsQuery().addIds(ids)).size(ids.length);
- try {
- SearchResponse response = client.search(searchRequest);
- healthChecker.health();
- return response;
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
+ return esClient.documents().mget(indexName, TYPE, ids);
}
- public void forceInsert(String indexName, String id, XContentBuilder source) throws IOException {
- IndexRequest request = (IndexRequest) prepareInsert(indexName, id, source);
- request.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
+ public Optional<Documents> ids(String indexName, String[] ids) {
+ return ids(indexName, Arrays.asList(ids));
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.ids(...)` indirectly reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.isExistsIndex(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.isExistsTemplate(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.retrievalIndexByAliases(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
}
- public SearchResponse search(String indexName, SearchSourceBuilder searchSourceBuilder) throws IOException {
+ public SearchResponse search(String indexName, Search search) {
indexName = formatIndexName(indexName);
- return doSearch(searchSourceBuilder, indexName);
+ return esClient.search(search, indexName);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.search(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
}
- public Map<String, Object> getIndex(String indexName) throws IOException {
+ public Optional<Index> getIndex(String indexName) {
if (StringUtil.isBlank(indexName)) {
- return new HashMap<>();
+ return Optional.empty();
}
indexName = formatIndexName(indexName);
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/" + indexName);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- return (Map<String, Object>) Optional.ofNullable(templates.get(indexName)).orElse(new HashMap<>());
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
- healthChecker.unHealth(e);
- throw e;
- } catch (IOException t) {
+ return esClient.index().get(indexName);
+ } catch (Exception t) {
healthChecker.unHealth(t);
throw t;
}
}
- public boolean createIndex(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping) throws IOException {
- indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- Gson gson = new Gson();
- request.settings(gson.toJson(settings), XContentType.JSON);
- request.mapping(TYPE, gson.toJson(mapping), XContentType.JSON);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
- }
-
- public List<String> retrievalIndexByAliases(String aliases) throws IOException {
+ public Collection<String> retrievalIndexByAliases(String aliases) {
aliases = formatIndexName(aliases);
- Response response;
- try {
- response = client.getLowLevelClient().performRequest(HttpGet.METHOD_NAME, "/_alias/" + aliases);
- healthChecker.health();
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) {
- Gson gson = new Gson();
- InputStreamReader reader;
- try {
- reader = new InputStreamReader(response.getEntity().getContent());
- } catch (Throwable t) {
- healthChecker.unHealth(t);
- throw t;
- }
- JsonObject responseJson = gson.fromJson(reader, JsonObject.class);
- log.debug("retrieval indexes by aliases {}, response is {}", aliases, responseJson);
- return new ArrayList<>(responseJson.keySet());
- }
- return Collections.emptyList();
+
+ return esClient.alias().indices(aliases).keySet();
}
/**
- * If your indexName is retrieved from elasticsearch through {@link #retrievalIndexByAliases(String)} or some other
- * method and it already contains namespace. Then you should delete the index by this method, this method will no
- * longer concatenate namespace.
+ * If your indexName is retrieved from elasticsearch through {@link
+ * #retrievalIndexByAliases(String)} or some other method and it already contains namespace.
+ * Then you should delete the index by this method, this method will no longer concatenate
+ * namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByIndexName(String indexName) throws IOException {
+ public boolean deleteByIndexName(String indexName) {
return deleteIndex(indexName, false);
}
/**
- * If your indexName is obtained from metadata or configuration and without namespace. Then you should delete the
- * index by this method, this method automatically concatenates namespace.
+ * If your indexName is obtained from metadata or configuration and without namespace. Then you
+ * should delete the index by this method, this method automatically concatenates namespace.
* <p>
* https://github.com/apache/skywalking/pull/3017
*/
- public boolean deleteByModelName(String modelName) throws IOException {
+ public boolean deleteByModelName(String modelName) {
return deleteIndex(modelName, true);
}
- protected boolean deleteIndex(String indexName, boolean formatIndexName) throws IOException {
+ protected boolean deleteIndex(String indexName, boolean formatIndexName) {
if (formatIndexName) {
indexName = formatIndexName(indexName);
}
- DeleteIndexRequest request = new DeleteIndexRequest(indexName);
- DeleteIndexResponse response;
- response = client.indices().delete(request);
- log.debug("delete {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().delete(indexName);
}
- public boolean isExistsIndex(String indexName) throws IOException {
+ public boolean isExistsIndex(String indexName) {
indexName = formatIndexName(indexName);
- GetIndexRequest request = new GetIndexRequest();
- request.indices(indexName);
- return client.indices().exists(request);
+
+ return esClient.index().exists(indexName);
}
- public Map<String, Object> getTemplate(String name) throws IOException {
+ public Optional<IndexTemplate> getTemplate(String name) {
name = formatIndexName(name);
+
try {
- Response response = client.getLowLevelClient()
- .performRequest(HttpGet.METHOD_NAME, "/_template/" + name);
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode != HttpStatus.SC_OK) {
- healthChecker.health();
- throw new IOException(
- "The response status code of template exists request should be 200, but it is " + statusCode);
- }
- Type type = new TypeToken<HashMap<String, Object>>() {
- }.getType();
- Map<String, Object> templates = new Gson().<HashMap<String, Object>>fromJson(
- new InputStreamReader(response.getEntity().getContent()),
- type
- );
- if (templates.containsKey(name)) {
- return (Map<String, Object>) templates.get(name);
- }
- return new HashMap<>();
- } catch (ResponseException e) {
- if (e.getResponse().getStatusLine().getStatusCode() == HttpStatus.SC_NOT_FOUND) {
- return new HashMap<>();
- }
+ return esClient.templates().get(name);
+ } catch (Exception e) {
healthChecker.unHealth(e);
throw e;
- } catch (IOException t) {
- healthChecker.unHealth(t);
- throw t;
}
}
- public boolean isExistsTemplate(String indexName) throws IOException {
+ public boolean isExistsTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient().performRequest(HttpHead.METHOD_NAME, "/_template/" + indexName);
-
- int statusCode = response.getStatusLine().getStatusCode();
- if (statusCode == HttpStatus.SC_OK) {
- return true;
- } else if (statusCode == HttpStatus.SC_NOT_FOUND) {
- return false;
- } else {
- throw new IOException(
- "The response status code of template exists request should be 200 or 404, but it is " + statusCode);
- }
+ return esClient.templates().exists(indexName);
}
public boolean createOrUpdateTemplate(String indexName, Map<String, Object> settings,
- Map<String, Object> mapping, int order) throws IOException {
+ Mappings mapping, int order) {
indexName = formatIndexName(indexName);
- String[] patterns = new String[] {indexName + "-*"};
-
- Map<String, Object> aliases = new HashMap<>();
- aliases.put(indexName, new JsonObject());
-
- Map<String, Object> template = new HashMap<>();
- template.put("index_patterns", patterns);
- template.put("aliases", aliases);
- template.put("settings", settings);
- template.put("mappings", mapping);
- template.put("order", order);
-
- HttpEntity entity = new NStringEntity(new Gson().toJson(template), ContentType.APPLICATION_JSON);
-
- Response response = client.getLowLevelClient()
- .performRequest(
- HttpPut.METHOD_NAME, "/_template/" + indexName, Collections.emptyMap(), entity);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+ return esClient.templates().createOrUpdate(indexName, settings, mapping, order);
}
- public boolean deleteTemplate(String indexName) throws IOException {
+ public boolean deleteTemplate(String indexName) {
indexName = formatIndexName(indexName);
- Response response = client.getLowLevelClient()
- .performRequest(HttpDelete.METHOD_NAME, "/_template/" + indexName);
- return response.getStatusLine().getStatusCode() == HttpStatus.SC_OK;
+
+ return esClient.templates().delete(indexName);
}
- public SearchResponse search(IndexNameMaker indexNameMaker,
- SearchSourceBuilder searchSourceBuilder) throws IOException {
- String[] indexNames = Arrays.stream(indexNameMaker.make()).map(this::formatIndexName).toArray(String[]::new);
- return doSearch(searchSourceBuilder, indexNames);
+ public SearchResponse search(IndexNameMaker indexNameMaker, Search search) {
+ final String[] indexNames =
+ Arrays.stream(indexNameMaker.make())
+ .map(this::formatIndexName)
+ .toArray(String[]::new);
+ return doSearch(search, indexNames);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.search(...)` indirectly reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.shutdown()` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##########
@@ -133,478 +91,248 @@ public ElasticSearchClient(String clusterNodes,
int socketTimeout) {
this.clusterNodes = clusterNodes;
this.protocol = protocol;
+ this.trustStorePath = trustStorePath;
+ this.trustStorePass = trustStorePass;
this.user = user;
this.password = password;
this.indexNameConverters = indexNameConverters;
- this.trustStorePath = trustStorePath;
- this.trustStorePass = trustStorePass;
this.connectTimeout = connectTimeout;
this.socketTimeout = socketTimeout;
}
@Override
- public void connect() throws IOException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException, CertificateException {
+ public void connect() {
connectLock.lock();
try {
- List<HttpHost> hosts = parseClusterNodes(protocol, clusterNodes);
- if (client != null) {
+ if (esClient != null) {
try {
- client.close();
+ esClient.close();
} catch (Throwable t) {
log.error("ElasticSearch client reconnection fails based on new config", t);
}
}
- client = createClient(hosts);
- client.ping();
+ esClient = org.apache.skywalking.library.elasticsearch.ElasticSearchClient
+ .builder()
+ .endpoints(clusterNodes.split(","))
+ .protocol(protocol)
+ .trustStorePath(trustStorePath)
+ .trustStorePass(trustStorePass)
+ .username(user)
+ .password(password)
+ .connectTimeout(connectTimeout)
+ .build();
+ esClient.connect();
} finally {
connectLock.unlock();
}
}
- protected RestHighLevelClient createClient(
- final List<HttpHost> pairsList) throws KeyStoreException, IOException, NoSuchAlgorithmException, CertificateException, KeyManagementException {
- RestClientBuilder builder;
- if (StringUtil.isNotEmpty(user) && StringUtil.isNotEmpty(password)) {
- final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
- credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(user, password));
-
- if (StringUtil.isEmpty(trustStorePath)) {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider));
- } else {
- KeyStore truststore = KeyStore.getInstance("jks");
- try (InputStream is = Files.newInputStream(Paths.get(trustStorePath))) {
- truststore.load(is, trustStorePass.toCharArray());
- }
- SSLContextBuilder sslBuilder = SSLContexts.custom().loadTrustMaterial(truststore, null);
- final SSLContext sslContext = sslBuilder.build();
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]))
- .setHttpClientConfigCallback(
- httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(
- credentialsProvider)
- .setSSLContext(sslContext));
- }
- } else {
- builder = RestClient.builder(pairsList.toArray(new HttpHost[0]));
- }
- builder.setRequestConfigCallback(
- requestConfigBuilder -> requestConfigBuilder
- .setConnectTimeout(connectTimeout)
- .setSocketTimeout(socketTimeout)
- );
-
- return new RestHighLevelClient(builder);
- }
-
@Override
- public void shutdown() throws IOException {
- client.close();
+ public void shutdown() {
+ esClient.close();
}
- public static List<HttpHost> parseClusterNodes(String protocol, String nodes) {
- List<HttpHost> httpHosts = new LinkedList<>();
- log.info("elasticsearch cluster nodes: {}", nodes);
- List<String> nodesSplit = Splitter.on(",").omitEmptyStrings().splitToList(nodes);
-
- for (String node : nodesSplit) {
- String host = node.split(":")[0];
- String port = node.split(":")[1];
- httpHosts.add(new HttpHost(host, Integer.parseInt(port), protocol));
- }
-
- return httpHosts;
+ public boolean createIndex(String indexName) {
+ return createIndex(indexName, null, null);
}
- public boolean createIndex(String indexName) throws IOException {
+ public boolean createIndex(String indexName,
+ Mappings mappings,
+ Map<String, ?> settings) {
indexName = formatIndexName(indexName);
- CreateIndexRequest request = new CreateIndexRequest(indexName);
- CreateIndexResponse response = client.indices().create(request);
- log.debug("create {} index finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+ return esClient.index().create(indexName, mappings, settings);
}
- public boolean updateIndexMapping(String indexName, Map<String, Object> mapping) throws IOException {
+ public boolean updateIndexMapping(String indexName, Mappings mapping) {
indexName = formatIndexName(indexName);
- Map<String, Object> properties = (Map<String, Object>) mapping.get(ElasticSearchClient.TYPE);
- PutMappingRequest putMappingRequest = new PutMappingRequest(indexName);
- Gson gson = new Gson();
- putMappingRequest.type(ElasticSearchClient.TYPE);
- putMappingRequest.source(gson.toJson(properties), XContentType.JSON);
- PutMappingResponse response = client.indices().putMapping(putMappingRequest);
- log.debug("put {} index mapping finished, isAcknowledged: {}", indexName, response.isAcknowledged());
- return response.isAcknowledged();
+
+ return esClient.index().putMapping(indexName, TYPE, mapping);
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `ElasticSearchClient.updateIndexMapping(...)` reads without synchronization from `this.esClient`. Potentially races with write in method `ElasticSearchClient.connect()`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: oap-server/server-alarm-plugin/pom.xml
##########
@@ -51,6 +51,10 @@
<groupId>org.mvel</groupId>
<artifactId>mvel2</artifactId>
</dependency>
+ <dependency>
Review comment:
*Critical OSS Vulnerability:*
### pkg:maven/org.apache.httpcomponents/httpclient
1 Critical, 4 Severe, 0 Moderate and 0 Unknown vulnerabilities have been found in a direct dependency
<!-- Lift_Details -->
<details>
<summary><b>CRITICAL Vulnerabilities (1)</b></summary>
<ul>
***
<details>
<summary>CVE-2013-4366</summary>
> #### [CVE-2013-4366] http/impl/client/HttpClientBuilder.java in Apache HttpClient 4.3.x before 4.3.1 ...
> http/impl/client/HttpClientBuilder.java in Apache HttpClient 4.3.x before 4.3.1 does not ensure that X509HostnameVerifier is not null, which allows attackers to have unspecified impact via vectors involving hostname verification.
>
> **CVSS Score:** 9.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
***
</ul>
</details>
<details>
<summary><b>SEVERE Vulnerabilities (4)</b></summary>
<ul>
***
<details>
<summary>CVE-2014-3577</summary>
> #### [CVE-2014-3577] org.apache.http.conn.ssl.AbstractVerifier in Apache HttpComponents HttpClient be...
> org.apache.http.conn.ssl.AbstractVerifier in Apache HttpComponents HttpClient before 4.3.5 and HttpAsyncClient before 4.0.2 does not properly verify that the server hostname matches a domain name in the subject's Common Name (CN) or subjectAltName field of the X.509 certificate, which allows man-in-the-middle attackers to spoof SSL servers via a "CN=" string in a field in the distinguished name (DN) of a certificate, as demonstrated by the "foo,CN=www.apache.org" string in the O field.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
***
<details>
<summary>CVE-2020-13956</summary>
> #### [CVE-2020-13956] Apache HttpClient versions prior to version 4.5.13 and 5.0.3 can misinterpret ma...
> Apache HttpClient versions prior to version 4.5.13 and 5.0.3 can misinterpret malformed authority component in request URIs passed to the library as java.net.URI object and pick the wrong target host for request execution.
>
> **CVSS Score:** 5.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:L/A:N
</details>
***
<details>
<summary>CVE-2015-5262</summary>
> #### [CVE-2015-5262] Resource Management Errors
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting during an SSL handshake, which allows remote attackers to cause a denial of service (HTTPS call hang) via unspecified vectors.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:N/A:P
</details>
***
<details>
<summary>CVE-2011-1498</summary>
> #### [CVE-2011-1498] Information Exposure
> Apache HttpClient 4.x before 4.1.1 in Apache HttpComponents, when used with an authenticating proxy server, sends the Proxy-Authorization header to the origin server, which allows remote web servers to obtain sensitive information by logging this header.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
***
</ul>
</details>
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#discussion_r702520333
##########
File path: oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/ElasticSearch.java
##########
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.skywalking.library.elasticsearch;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.linecorp.armeria.client.ClientFactory;
+import com.linecorp.armeria.client.WebClient;
+import com.linecorp.armeria.client.WebClientBuilder;
+import com.linecorp.armeria.client.endpoint.EndpointGroup;
+import com.linecorp.armeria.client.logging.LoggingClient;
+import com.linecorp.armeria.common.HttpData;
+import com.linecorp.armeria.common.HttpStatus;
+import com.linecorp.armeria.common.SessionProtocol;
+import com.linecorp.armeria.common.auth.BasicToken;
+import com.linecorp.armeria.common.util.Exceptions;
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.util.StringUtil;
+import org.apache.skywalking.library.elasticsearch.bulk.BulkProcessorBuilder;
+import org.apache.skywalking.library.elasticsearch.client.AliasClient;
+import org.apache.skywalking.library.elasticsearch.client.DocumentClient;
+import org.apache.skywalking.library.elasticsearch.client.IndexClient;
+import org.apache.skywalking.library.elasticsearch.client.SearchClient;
+import org.apache.skywalking.library.elasticsearch.client.TemplateClient;
+import org.apache.skywalking.library.elasticsearch.requests.search.Search;
+import org.apache.skywalking.library.elasticsearch.response.NodeInfo;
+import org.apache.skywalking.library.elasticsearch.response.search.SearchResponse;
+
+@Slf4j
+public final class ElasticSearch implements Closeable {
+ private final ObjectMapper mapper = new ObjectMapper()
+ .setSerializationInclusion(JsonInclude.Include.NON_NULL)
+ .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
+
+ private final WebClient client;
+
+ private final ClientFactory clientFactory;
+
+ private final CompletableFuture<ElasticSearchVersion> version;
+
+ private final TemplateClient templateClient;
+
+ private final IndexClient indexClient;
+
+ private final DocumentClient documentClient;
+
+ private final AliasClient aliasClient;
+
+ private final SearchClient searchClient;
+
+ ElasticSearch(SessionProtocol protocol,
+ String username, String password,
+ EndpointGroup endpointGroup,
+ ClientFactory clientFactory) {
+ this.clientFactory = clientFactory;
+ this.version = new CompletableFuture<>();
+
+ final WebClientBuilder builder =
+ WebClient.builder(protocol, endpointGroup)
+ .factory(clientFactory)
+ .decorator(
+ LoggingClient.builder()
+ .logger(log)
+ .newDecorator());
Review comment:
Added configuration
##########
File path: oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/bulk/BulkProcessor.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.skywalking.library.elasticsearch.bulk;
+
+import com.linecorp.armeria.client.WebClient;
+import com.linecorp.armeria.common.HttpStatus;
+import com.linecorp.armeria.common.util.Exceptions;
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import lombok.SneakyThrows;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.library.elasticsearch.ElasticSearchVersion;
+import org.apache.skywalking.library.elasticsearch.requests.IndexRequest;
+import org.apache.skywalking.library.elasticsearch.requests.UpdateRequest;
+import org.apache.skywalking.library.elasticsearch.requests.factory.RequestFactory;
+
+import static java.util.Objects.requireNonNull;
+
+@Slf4j
+public final class BulkProcessor {
+ private final ArrayBlockingQueue<Object> requests;
+
+ private final CompletableFuture<ElasticSearchVersion> version;
+ private final WebClient client;
+
+ private final int bulkActions;
+ private final int concurrentRequests;
+
+ public BulkProcessor(
+ final CompletableFuture<ElasticSearchVersion> version,
+ final WebClient client, final int bulkActions,
+ final Duration flushInterval, final int concurrentRequests) {
+ this.version = requireNonNull(version, "requestFactory");
+ this.client = requireNonNull(client, "client");
+ this.bulkActions = bulkActions;
+
+ requireNonNull(flushInterval, "flushInterval");
+
+ this.concurrentRequests = concurrentRequests;
+ this.requests = new ArrayBlockingQueue<>(bulkActions + 1);
+ Executors.newSingleThreadScheduledExecutor(r -> {
Review comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on pull request #7634: Rebuilt ElasticSearch client on top of their REST API
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #7634:
URL: https://github.com/apache/skywalking/pull/7634#issuecomment-913338047
> I tested locally, all looks great to me except `EmptyEndpointGroupException` as follows when `es` is shutdown and connected failures:
>
> ```shell
> 2021-09-06 11:14:10,020 org.apache.skywalking.library.elasticsearch.ElasticSearch 149 [armeria-eventloop-nio-4-2] DEBUG [] - [creqId=3fe1891d, preqId=8179d41f][http://UNKNOWN/#GET] Request: {startTime=2021-09-06T03:14:10.010Z(1630898050010987), length=0B, duration=4538ยตs(4538520ns), cause=com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException, scheme=none+http, name=GET, headers=[:method=GET, :path=/, :scheme=http, :authority=UNKNOWN]}
> 2021-09-06 11:14:10,021 org.apache.skywalking.library.elasticsearch.ElasticSearch 186 [armeria-eventloop-nio-4-2] WARN [] - [creqId=3fe1891d, preqId=8179d41f][http://UNKNOWN/#GET] Response: {startTime=2021-09-06T03:14:10.020Z(1630898050020718), length=0B, duration=0ns, totalDuration=9749ยตs(9749393ns), cause=com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException, headers=[:status=0]}
> com.linecorp.armeria.client.UnprocessedRequestException: com.linecorp.armeria.client.endpoint.EmptyEndpointGroupException
> ```
>
> I think `EmptyEndpointGroupException` is not enough clear to point out the cause of the error?
There is warning / error message before what you posted, also, it has health check registered .
```
2021-09-06 12:52:21,329 org.apache.skywalking.oap.server.telemetry.api.HealthCheckMetrics 50 [armeria-common-worker-nio-2-1] WARN [] - Health check fails. reason: No healthy endpoint
2021-09-06 12:52:21,330 org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient 146 [armeria-common-worker-nio-2-1] ERROR [] - Failed to recreate ElasticSearch client based on new config
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org