You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2021/09/22 08:07:02 UTC
[GitHub] [skywalking] LIU-WEI-git opened a new pull request #7766: IoTDB storage plugin
LIU-WEI-git opened a new pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766
Hi, I am a participant in [Summer 2021 of Open Source Promotion Plan](https://summer.iscas.ac.cn/#/homepage). [Here](https://summer.iscas.ac.cn/#/org/prodetail/210070771) is my project homepage. Now I have finished my project under the guidance of @jixuan1989 from IoTDB and @wu-sheng from Skywalking. However, I still have much work to do. This PR will help me find more bug and deficiencies of my work.
[Here](https://github.com/LIU-WEI-git/skywalking/tree/iotdb-adapter/oap-server/server-storage-plugin/storage-iotdb-plugin) is the document for this project. It is written in Chinese.
This is a large module for Skywalking. I wrote more than 5 thousand source code lines and comment lines. They contains the core code and unit tests. When I wrote the code, I took the other storage plugin as a reference.
IoTDB doesn't have the index concept. In order to index some fields, I store the value of the index as a node of the storage path. Those nodes in storage path will be stored in the memory, so they serve as index. Eventually the data from the same table is spread across multiple devices. But IoTDB does not support cross-device query well. When I need use aggregation query (e.g. `max`, `avg`, `sum`), sort query (e.g. `order by`) or pagination query (e.g. `limit`, `offset`), I have to use a violence method that query all data meets the condition and then sort, aggregate or paginate them.
Although I have run and tested this IoTDB storage plugin locally, I know it is not enough and it need e2e tests at least. Hope you review my work and give me some suggestions.
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/CHANGES.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715279768
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
Review comment:
I have set the `tomcat-embed-core` dependency version to 10.0.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753744093
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
OK, @LIU-WEI-git please recheck, I used to notice this jar in previous dependency check report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753775904
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBInsertRequest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+import org.apache.skywalking.oap.server.library.client.request.InsertRequest;
+import org.apache.skywalking.oap.server.library.client.request.UpdateRequest;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+
+@Getter
+@Setter
+@ToString
+@Slf4j
+public class IoTDBInsertRequest implements InsertRequest, UpdateRequest {
+ private String modelName;
+ private long time;
+ private List<String> indexes;
+ private List<String> indexValues;
+ private List<String> measurements;
+ private List<TSDataType> measurementTypes;
+ private List<Object> measurementValues;
+
+ public <T extends StorageData> IoTDBInsertRequest(String modelName, long time, T storageData,
+ StorageHashMapBuilder<T> storageBuilder) {
+ this.modelName = modelName;
+ this.time = time;
+ indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexValues = new ArrayList<>(indexes.size());
+ Map<String, Object> storageMap = storageBuilder.entity2Storage(storageData);
+
+ indexes.forEach(index -> {
+ if (index.equals(IoTDBClient.ID_IDX)) {
+ indexValues.add(storageData.id());
+ } else if (storageMap.containsKey(index)) {
+ indexValues.add(String.valueOf(storageMap.get(index)));
+ storageMap.remove(index);
+ }
+ });
+
+ // time_bucket has changed to time before calling this method, so remove it from measurements
+ storageMap.remove(IoTDBClient.TIME_BUCKET);
+ // processing illegal value
+ Iterator<Map.Entry<String, Object>> entryIterator = storageMap.entrySet().iterator();
+ while (entryIterator.hasNext()) {
+ Map.Entry<String, Object> entry = entryIterator.next();
+ if (entry.getValue() == null) {
+ entryIterator.remove();
+ }
+ if (entry.getValue() instanceof StorageDataComplexObject) {
+ storageMap.put(entry.getKey(), ((StorageDataComplexObject) entry.getValue()).toStorageData());
+ }
+ }
+
+ measurements = new ArrayList<>(storageMap.keySet());
+ Map<String, TSDataType> columnAndTypeMap = IoTDBTableMetaInfo.get(modelName).getColumnAndTypeMap();
+ measurementTypes = new ArrayList<>(measurements.size());
+ for (String measurement : measurements) {
+ measurementTypes.add(columnAndTypeMap.get(measurement));
+ }
+ measurementValues = new ArrayList<>(storageMap.values());
+
+ // processing illegal measurement
Review comment:
What do you mean illegal? I think this is a wrong word chosen? Maybe you mean to process several special fields which formats unsupported by IoTDB?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753743522
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
Check the previous exclusive, the tricky thing here is, this repo changed the artifact ID. This is manual override dependency, and keep consistent with our existing dependency.
But the version should be managed in the bom.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
OK, @LIU-WEI-git please recheck, I used to notice this jar in previous dependency check report.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
@kezhenxu94 I can see this now.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -44,6 +44,16 @@
<dependency>
<groupId>org.apache.iotdb</groupId>
<artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
Review comment:
@jixuan1989 We manually replaced this with `1.6.0` to avoid 2 versions existing together in the final binary.
##########
File path: docs/en/setup/backend/backend-storage.md
##########
@@ -280,6 +280,24 @@ storage:
All connection-related settings, including URL link, username, and password are found in `application.yml`.
Only part of the settings are listed here. Please follow [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool document for full settings.
+## IoTDB
Review comment:
I think you missed the option list on the top of file.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBTableInstaller.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelInstaller;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+
+public class IoTDBTableInstaller extends ModelInstaller {
Review comment:
Could you consider to add a long comment here about what are typically tables look like? Such as including 5 types
1. Metadata, use `service_traffic` for example.
2. Metrics, use `service_cpm`
3. Trace segment
4. Log
5. Profiling snapshots
With all these, when we face bugs one day, or with IoTDB improvement, we will know the expected table structure.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBHistoryDeleteDAO.java
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBHistoryDeleteDAO implements IHistoryDeleteDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public void deleteHistory(Model model, String timeBucketColumnName, int ttl) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("TTL execution log, model: {}, TTL: {}", model.getName(), ttl);
+ }
Review comment:
Do we really need logs like this? Even we need, I think we should move this to general DataTTLKeeperTimer, rather than implementation. Feel free to raise another PR to add this if needed.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBInsertRequest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+import org.apache.skywalking.oap.server.library.client.request.InsertRequest;
+import org.apache.skywalking.oap.server.library.client.request.UpdateRequest;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+
+@Getter
+@Setter
+@ToString
+@Slf4j
+public class IoTDBInsertRequest implements InsertRequest, UpdateRequest {
+ private String modelName;
+ private long time;
+ private List<String> indexes;
+ private List<String> indexValues;
+ private List<String> measurements;
+ private List<TSDataType> measurementTypes;
+ private List<Object> measurementValues;
+
+ public <T extends StorageData> IoTDBInsertRequest(String modelName, long time, T storageData,
+ StorageHashMapBuilder<T> storageBuilder) {
+ this.modelName = modelName;
+ this.time = time;
+ indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexValues = new ArrayList<>(indexes.size());
+ Map<String, Object> storageMap = storageBuilder.entity2Storage(storageData);
+
+ indexes.forEach(index -> {
+ if (index.equals(IoTDBClient.ID_IDX)) {
+ indexValues.add(storageData.id());
+ } else if (storageMap.containsKey(index)) {
+ indexValues.add(String.valueOf(storageMap.get(index)));
+ storageMap.remove(index);
+ }
+ });
+
+ // time_bucket has changed to time before calling this method, so remove it from measurements
+ storageMap.remove(IoTDBClient.TIME_BUCKET);
+ // processing illegal value
+ Iterator<Map.Entry<String, Object>> entryIterator = storageMap.entrySet().iterator();
+ while (entryIterator.hasNext()) {
+ Map.Entry<String, Object> entry = entryIterator.next();
+ if (entry.getValue() == null) {
+ entryIterator.remove();
+ }
+ if (entry.getValue() instanceof StorageDataComplexObject) {
+ storageMap.put(entry.getKey(), ((StorageDataComplexObject) entry.getValue()).toStorageData());
+ }
+ }
+
+ measurements = new ArrayList<>(storageMap.keySet());
+ Map<String, TSDataType> columnAndTypeMap = IoTDBTableMetaInfo.get(modelName).getColumnAndTypeMap();
+ measurementTypes = new ArrayList<>(measurements.size());
+ for (String measurement : measurements) {
+ measurementTypes.add(columnAndTypeMap.get(measurement));
+ }
+ measurementValues = new ArrayList<>(storageMap.values());
+
+ // processing illegal measurement
Review comment:
What do you mean illegal? I think this is a wrong word chosen? Maybe you mean to process several special fields which formats unsupported by IoTDB?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r756639940
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBAggregationQueryDAO.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.input.TopNCondition;
+import org.apache.skywalking.oap.server.core.query.type.KeyValue;
+import org.apache.skywalking.oap.server.core.query.type.SelectedRecord;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBAggregationQueryDAO implements IAggregationQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<SelectedRecord> sortMetrics(TopNCondition condition, String valueColumnName, Duration duration,
+ List<KeyValue> additionalConditions) throws IOException {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without aggregation function.
+ // https://github.com/apache/iotdb/issues/4006
+ StringBuilder query = new StringBuilder();
+ query.append(String.format("select %s from ", valueColumnName));
+ query = client.addModelPath(query, condition.getName());
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(duration.getStartTimestamp())
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(duration.getStartTimestamp());
+ if (additionalConditions != null) {
+ for (KeyValue additionalCondition : additionalConditions) {
+ query.append(" and ").append(additionalCondition.getKey()).append(" = \"")
+ .append(additionalCondition.getValue()).append("\"");
+ }
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<SelectedRecord> topEntities = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+
+ Map<String, Double> entityIdAndSumMap = new HashMap<>();
+ Map<String, Integer> entityIdAndCountMap = new HashMap<>();
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ double value = Double.parseDouble(fields.get(1).getStringValue());
+ entityIdAndSumMap.merge(entityId, value, Double::sum);
+ entityIdAndCountMap.merge(entityId, 1, Integer::sum);
+ }
+
+ entityIdAndSumMap.forEach((String entityId, Double sum) -> {
+ double count = entityIdAndCountMap.get(entityId);
+ double avg = sum / count;
+ SelectedRecord topNEntity = new SelectedRecord();
+ topNEntity.setId(entityId);
+ topNEntity.setValue(String.valueOf(avg));
Review comment:
ok, thanks you for pointing out.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBAggregationQueryDAO.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.input.TopNCondition;
+import org.apache.skywalking.oap.server.core.query.type.KeyValue;
+import org.apache.skywalking.oap.server.core.query.type.SelectedRecord;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBAggregationQueryDAO implements IAggregationQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<SelectedRecord> sortMetrics(TopNCondition condition, String valueColumnName, Duration duration,
+ List<KeyValue> additionalConditions) throws IOException {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without aggregation function.
+ // https://github.com/apache/iotdb/issues/4006
+ StringBuilder query = new StringBuilder();
+ query.append(String.format("select %s from ", valueColumnName));
+ query = client.addModelPath(query, condition.getName());
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(duration.getStartTimestamp())
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(duration.getStartTimestamp());
+ if (additionalConditions != null) {
+ for (KeyValue additionalCondition : additionalConditions) {
+ query.append(" and ").append(additionalCondition.getKey()).append(" = \"")
+ .append(additionalCondition.getValue()).append("\"");
+ }
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<SelectedRecord> topEntities = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+
+ Map<String, Double> entityIdAndSumMap = new HashMap<>();
+ Map<String, Integer> entityIdAndCountMap = new HashMap<>();
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ double value = Double.parseDouble(fields.get(1).getStringValue());
+ entityIdAndSumMap.merge(entityId, value, Double::sum);
+ entityIdAndCountMap.merge(entityId, 1, Integer::sum);
+ }
+
+ entityIdAndSumMap.forEach((String entityId, Double sum) -> {
+ double count = entityIdAndCountMap.get(entityId);
+ double avg = sum / count;
+ SelectedRecord topNEntity = new SelectedRecord();
+ topNEntity.setId(entityId);
+ topNEntity.setValue(String.valueOf(avg));
Review comment:
ok, thank you for pointing out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-924696334
ok, I will deal with them later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-925707171
> I will package a IoTDB dev docker image later.
SkyWalking is building dev image automatically for every default branch commit, using commitid as tag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714636106
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBMetricsDAOTest.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.skywalking.oap.server.core.analysis.DownSampling;
+import org.apache.skywalking.oap.server.core.analysis.FunctionCategory;
+import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic;
+import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.model.ExtraQueryIndex;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBStorageConfig;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+import static org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClientTest.retrieval;
+
+public class IoTDBMetricsDAOTest {
Review comment:
I need this method to build `org.apache.skywalking.oap.server.core.storage.model`, and add it to `IoTDBTableMetaInfo.TABLE_META_INFOS`. When those query tests run, they need to get information from `IoTDBTableMetaInfo`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713742997
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/README.md
##########
@@ -0,0 +1,194 @@
+Apache IoTDB - Apache Skywalking适配器
Review comment:
Chinese document is not recommended to host in SkyWalking codebase.
Our official doc is here, https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md. And you could read the latest from https://skywalking.apache.org/docs/main/latest/en/setup/backend/backend-storage/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-924689861
@dmsolr Please lead on review, due to you lead on InfluxDB storage implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714939178
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBBrowserLogQueryDAO.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.util.StringUtil;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.browser.source.BrowserErrorCategory;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLog;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLogs;
+import org.apache.skywalking.oap.server.core.query.type.ErrorCategory;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBBrowserLogQueryDAO implements IBrowserLogQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<BrowserErrorLogRecord> storageBuilder = new BrowserErrorLogRecord.Builder();
+
+ @Override
+ public BrowserErrorLogs queryBrowserErrorLogs(String serviceId, String serviceVersionId, String pagePathId,
+ BrowserErrorCategory category, long startSecondTB,
+ long endSecondTB, int limit, int from) throws IOException {
+ StringBuilder query = new StringBuilder();
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query.append("select * from ");
+ query = client.addModelPath(query, BrowserErrorLogRecord.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ if (StringUtil.isNotEmpty(serviceId)) {
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ }
+ query = client.addQueryIndexValue(BrowserErrorLogRecord.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where 1=1");
+ if (startSecondTB != 0 && endSecondTB != 0) {
+ query.append(" and ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startSecondTB));
+ query.append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endSecondTB));
+ }
+ if (StringUtil.isNotEmpty(serviceVersionId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.SERVICE_VERSION_ID).append(" = \"").append(serviceVersionId).append("\"");
+ }
+ if (StringUtil.isNotEmpty(pagePathId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.PAGE_PATH_ID).append(" = \"").append(pagePathId).append("\"");
+ }
+ if (Objects.nonNull(category)) {
+ query.append(" and ").append(BrowserErrorLogRecord.ERROR_CATEGORY).append(" = ").append(category.getValue());
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString();
+ queryString = queryString.replace("1=1 and ", "");
+
+ List<? super StorageData> storageDataList = client.filterQuery(BrowserErrorLogRecord.INDEX_NAME, queryString, storageBuilder);
+ List<BrowserErrorLogRecord> browserErrorLogRecordList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> browserErrorLogRecordList.add((BrowserErrorLogRecord) storageData));
+ // resort by self, because of the select query result order by time.
+ browserErrorLogRecordList.sort((BrowserErrorLogRecord b1, BrowserErrorLogRecord b2) -> Long.compare(b2.getTimestamp(), b1.getTimestamp()));
+ BrowserErrorLogs logs = new BrowserErrorLogs();
+ int limitCount = 0;
+ for (int i = 0; i < browserErrorLogRecordList.size(); i++) {
+ if (i >= from && limitCount < limit) {
+ limitCount++;
+ BrowserErrorLogRecord record = browserErrorLogRecordList.get(i);
+ if (CollectionUtils.isNotEmpty(record.getDataBinary())) {
+ BrowserErrorLog log = iotdbParserDataBinary(record.getDataBinary());
+ logs.getLogs().add(log);
+ }
+ }
+ }
+ logs.setTotal(storageDataList.size());
+ return logs;
+ }
+
+ private BrowserErrorLog iotdbParserDataBinary(byte[] dataBinaryBase64) {
+ try {
+ BrowserErrorLog log = new BrowserErrorLog();
+ org.apache.skywalking.apm.network.language.agent.v3.BrowserErrorLog browserErrorLog = org.apache.skywalking.apm.network.language.agent.v3.BrowserErrorLog
+ .parseFrom(dataBinaryBase64);
+ log.setService(browserErrorLog.getService());
+ log.setServiceVersion(browserErrorLog.getServiceVersion());
+ log.setTime(browserErrorLog.getTime());
+ log.setPagePath(browserErrorLog.getPagePath());
+ log.setCategory(ErrorCategory.valueOf(browserErrorLog.getCategory().name().toUpperCase()));
+ log.setGrade(browserErrorLog.getGrade());
+ log.setMessage(browserErrorLog.getMessage());
+ log.setLine(browserErrorLog.getLine());
+ log.setCol(browserErrorLog.getCol());
+ log.setStack(browserErrorLog.getStack());
+ log.setErrorUrl(browserErrorLog.getErrorUrl());
+ log.setFirstReportedError(browserErrorLog.getFirstReportedError());
+
+ return log;
+ } catch (InvalidProtocolBufferException e) {
Review comment:
Is this an IOException? If so, we don't need this try/catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715280541
##########
File path: dist-material/release-docs/LICENSE
##########
@@ -327,6 +327,7 @@ The text of each license is the standard Apache 2.0 license.
Armeria 1.11.0, http://github.com/line/armeria, Apache 2.0
Brotli4j 1.6.0, https://github.com/hyperxpro/Brotli4j, Apache 2.0
micrometer 1.7.3, https://github.com/micrometer-metrics/micrometer, Apache 2.0
+ iotdb-session 0.12.2, https://github.com/apache/iotdb, Apache 2.0
Review comment:
All dependency libs should be added here and https://github.com/apache/skywalking/blob/master/tools/dependencies/known-oap-backend-dependencies.txt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753737657
##########
File path: dist-material/release-docs/LICENSE
##########
@@ -394,6 +399,8 @@ The following components are provided under the EPL License. See project link fo
The text of each license is also included at licenses/LICENSE-[project].txt.
logback 1.1.11: https://github.com/qos-ch/logback: EPL 1.0
+ logback-classic 1.2.3: https://github.com/qos-ch/logback: EPL 1.0
Review comment:
Can you please also update the version in the previous line (401) 👆 , it should be 1.2.3 as well
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBStorageProvider.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.storage.IBatchDAO;
+import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
+import org.apache.skywalking.oap.server.core.storage.StorageDAO;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.StorageModule;
+import org.apache.skywalking.oap.server.core.storage.cache.INetworkAddressAliasDAO;
+import org.apache.skywalking.oap.server.core.storage.management.UITemplateManagementDAO;
+import org.apache.skywalking.oap.server.core.storage.model.ModelCreator;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileTaskLogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileTaskQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IAlarmQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ILogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IMetadataQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IMetricsQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITopNRecordsQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
+import org.apache.skywalking.oap.server.library.module.ModuleConfig;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+import org.apache.skywalking.oap.server.library.module.ModuleProvider;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBBatchDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBStorageDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.cache.IoTDBNetworkAddressAliasDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.management.IoTDBUITemplateManagementDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileTaskLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileTaskQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBAlarmQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBMetadataQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBMetricsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTopNRecordsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTraceQueryDAO;
+import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
+import org.apache.skywalking.oap.server.telemetry.api.HealthCheckMetrics;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsCreator;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsTag;
+
+import java.io.IOException;
+
+@Slf4j
+public class IoTDBStorageProvider extends ModuleProvider {
+ private final IoTDBStorageConfig config;
+ private IoTDBClient client;
+
+ public IoTDBStorageProvider() {
+ config = new IoTDBStorageConfig();
+ }
+
+ @Override
+ public String name() {
+ return "iotdb";
+ }
+
+ @Override
+ public Class<? extends ModuleDefine> module() {
+ return StorageModule.class;
+ }
+
+ @Override
+ public ModuleConfig createConfigBeanIfAbsent() {
+ return config;
+ }
+
+ @Override
+ public void prepare() throws ServiceNotProvidedException {
+ this.registerServiceImplementation(StorageBuilderFactory.class, new StorageBuilderFactory.Default());
+
+ try {
+ client = new IoTDBClient(config);
+ } catch (IOException e) {
+ e.printStackTrace();
Review comment:
Never do this in production code, @codeglzhang will you be able to add this prohibition any soon? As we discussed in the other PR
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String GROUP_IDX = "service_group";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) throws IOException {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug(request.toString());
Review comment:
Can you add some descriptive message to the log so that we can search them when needed, such as
```suggestion
log.debug("Writing data to IoTDB: {}", request);
```
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
I don't see there is `lz4` in `iotdb-session`'s dependency tree, please recheck
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBInsertRequest.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+import org.apache.skywalking.oap.server.library.client.request.InsertRequest;
+import org.apache.skywalking.oap.server.library.client.request.UpdateRequest;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+
+@Getter
+@Setter
+@Slf4j
+public class IoTDBInsertRequest implements InsertRequest, UpdateRequest {
+ private String modelName;
+ private long time;
+ private List<String> indexes;
+ private List<String> indexValues;
+ private List<String> measurements;
+ private List<TSDataType> measurementTypes;
+ private List<Object> measurementValues;
+
+ public <T extends StorageData> IoTDBInsertRequest(String modelName, long time, T storageData,
+ StorageHashMapBuilder<T> storageBuilder) {
+ this.modelName = modelName;
+ this.time = time;
+ indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexValues = new ArrayList<>(indexes.size());
+ Map<String, Object> storageMap = storageBuilder.entity2Storage(storageData);
+
+ indexes.forEach(index -> {
+ if (index.equals(IoTDBClient.ID_IDX)) {
+ indexValues.add(storageData.id());
+ } else if (storageMap.containsKey(index)) {
+ indexValues.add(String.valueOf(storageMap.get(index)));
+ storageMap.remove(index);
+ }
+ });
+
+ // time_bucket has changed to time before calling this method, so remove it from measurements
+ storageMap.remove(IoTDBClient.TIME_BUCKET);
+ // processing illegal value
+ Iterator<Map.Entry<String, Object>> entryIterator = storageMap.entrySet().iterator();
+ while (entryIterator.hasNext()) {
+ Map.Entry<String, Object> entry = entryIterator.next();
+ if (entry.getValue() == null) {
+ entryIterator.remove();
+ }
+ if (entry.getValue() instanceof StorageDataComplexObject) {
+ storageMap.put(entry.getKey(), ((StorageDataComplexObject) entry.getValue()).toStorageData());
+ }
+ }
+
+ measurements = new ArrayList<>(storageMap.keySet());
+ Map<String, TSDataType> columnAndTypeMap = IoTDBTableMetaInfo.get(modelName).getColumnAndTypeMap();
+ measurementTypes = new ArrayList<>(measurements.size());
+ for (String measurement : measurements) {
+ measurementTypes.add(columnAndTypeMap.get(measurement));
+ }
+ measurementValues = new ArrayList<>(storageMap.values());
+
+ // processing illegal measurement
+ for (String key : storageMap.keySet()) {
+ if (key.equals(IoTDBClient.TIMESTAMP) || key.contains(".")) {
+ int idx = measurements.indexOf(key);
+ measurements.set(idx, "\"" + key + "\"");
+ }
+ }
+ }
+
+ @Override
+ public String toString() {
Review comment:
Use `@lombok.ToString` instead
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
Review comment:
```suggestion
```
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBTableMetaInfo.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import com.google.gson.JsonObject;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Getter;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.analysis.NodeType;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+@Getter
+@Builder
+@AllArgsConstructor
+public class IoTDBTableMetaInfo {
+ private static final Map<String, IoTDBTableMetaInfo> TABLE_META_INFOS = new HashMap<>();
+
+ private final Model model;
+ private final Map<String, TSDataType> columnAndTypeMap;
+ private final List<String> indexes;
+
+ public static void addModel(Model model) {
+ final List<ModelColumn> columns = model.getColumns();
+ final Map<String, String> storageAndIndexMap = new HashMap<>();
+ final Map<String, TSDataType> columnAndTypeMap = new HashMap<>();
+ final List<String> indexes = new ArrayList<>();
+
+ storageAndIndexMap.put(model.getName(), IoTDBClient.ID_IDX);
+ columns.forEach(column -> {
+ String columnName = column.getColumnName().getName();
+ if (columnName.equals(IoTDBClient.ENTITY_ID_IDX) || columnName.equals(IoTDBClient.NODE_TYPE_IDX) ||
+ columnName.equals(IoTDBClient.SERVICE_ID_IDX) || columnName.equals(IoTDBClient.GROUP_IDX) ||
+ columnName.equals(IoTDBClient.TRACE_ID_IDX)) {
+ storageAndIndexMap.put(column.getColumnName().getStorageName(), columnName);
+ } else {
+ columnAndTypeMap.put(columnName, typeToTSDataType(column.getType()));
+ }
+ });
+
+ // TODO optimize map to list
Review comment:
Seems this is done?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753763797
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBStorageProvider.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.storage.IBatchDAO;
+import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
+import org.apache.skywalking.oap.server.core.storage.StorageDAO;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.StorageModule;
+import org.apache.skywalking.oap.server.core.storage.cache.INetworkAddressAliasDAO;
+import org.apache.skywalking.oap.server.core.storage.management.UITemplateManagementDAO;
+import org.apache.skywalking.oap.server.core.storage.model.ModelCreator;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileTaskLogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileTaskQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IAlarmQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ILogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IMetadataQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IMetricsQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITopNRecordsQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
+import org.apache.skywalking.oap.server.library.module.ModuleConfig;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+import org.apache.skywalking.oap.server.library.module.ModuleProvider;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBBatchDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBStorageDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.cache.IoTDBNetworkAddressAliasDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.management.IoTDBUITemplateManagementDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileTaskLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileTaskQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBAlarmQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBMetadataQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBMetricsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTopNRecordsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTraceQueryDAO;
+import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
+import org.apache.skywalking.oap.server.telemetry.api.HealthCheckMetrics;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsCreator;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsTag;
+
+import java.io.IOException;
+
+@Slf4j
+public class IoTDBStorageProvider extends ModuleProvider {
+ private final IoTDBStorageConfig config;
+ private IoTDBClient client;
+
+ public IoTDBStorageProvider() {
+ config = new IoTDBStorageConfig();
+ }
+
+ @Override
+ public String name() {
+ return "iotdb";
+ }
+
+ @Override
+ public Class<? extends ModuleDefine> module() {
+ return StorageModule.class;
+ }
+
+ @Override
+ public ModuleConfig createConfigBeanIfAbsent() {
+ return config;
+ }
+
+ @Override
+ public void prepare() throws ServiceNotProvidedException {
+ this.registerServiceImplementation(StorageBuilderFactory.class, new StorageBuilderFactory.Default());
+
+ try {
+ client = new IoTDBClient(config);
+ } catch (IOException e) {
+ e.printStackTrace();
Review comment:
ok, I have fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753771734
##########
File path: docs/en/setup/backend/backend-storage.md
##########
@@ -280,6 +280,24 @@ storage:
All connection-related settings, including URL link, username, and password are found in `application.yml`.
Only part of the settings are listed here. Please follow [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool document for full settings.
+## IoTDB
Review comment:
I think you missed the option list on the top of file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-979590873
Waiting to TopN verification. @LIU-WEI-git Please confirm when you are done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r742503228
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,17 +61,26 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.13.0-20211102.115555-1.jar
+iotdb-thrift-0.13.0-20211102.115555-1.jar
Review comment:
This will be updated after the official release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753743990
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
I mean there is no `lz4`, even `net.jpountz.lz4:lz4`, in `iotdb-session`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r754864618
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBHistoryDeleteDAO.java
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBHistoryDeleteDAO implements IHistoryDeleteDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public void deleteHistory(Model model, String timeBucketColumnName, int ttl) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("TTL execution log, model: {}, TTL: {}", model.getName(), ttl);
+ }
Review comment:
Influxdb storage plugin has this log as well. I think we could keep it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713765699
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/README.md
##########
@@ -0,0 +1,194 @@
+Apache IoTDB - Apache Skywalking适配器
Review comment:
If IoTDB storage plugin could work well. I will edit official doc later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713752550
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBEventQueryDAO.java
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import org.apache.skywalking.oap.server.core.query.PaginationUtils;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.type.event.EventQueryCondition;
+import org.apache.skywalking.oap.server.core.query.type.event.EventType;
+import org.apache.skywalking.oap.server.core.query.type.event.Events;
+import org.apache.skywalking.oap.server.core.query.type.event.Source;
+import org.apache.skywalking.oap.server.core.source.Event;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+public class IoTDBEventQueryDAO implements IEventQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<Event> storageBuilder = new Event.Builder();
+
+ public IoTDBEventQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public Events queryEvents(EventQueryCondition condition) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(condition, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = 0; i < storageDataList.size(); i++) {
+ if (i >= page.getFrom() && limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ org.apache.skywalking.oap.server.core.query.type.event.Event resultEvent = parseEvent(event);
+ events.getEvents().add(resultEvent);
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
+ (org.apache.skywalking.oap.server.core.query.type.event.Event e1,
+ org.apache.skywalking.oap.server.core.query.type.event.Event e2)
+ -> Long.compare(e2.getStartTime(), e1.getStartTime()));
+ } else {
+ events.getEvents().sort(
+ Comparator.comparingLong(org.apache.skywalking.oap.server.core.query.type.event.Event::getStartTime));
+ }
+ return events;
+ }
+
+ @Override
+ public Events queryEvents(List<EventQueryCondition> conditionList) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(conditionList, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ EventQueryCondition condition = conditionList.get(0);
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = 0; i < storageDataList.size(); i++) {
+ if (i >= page.getFrom() && limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ org.apache.skywalking.oap.server.core.query.type.event.Event resultEvent = parseEvent(event);
+ events.getEvents().add(resultEvent);
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
Review comment:
It has been initialized in constructor and won't be null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714663981
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/README.md
##########
@@ -0,0 +1,194 @@
+Apache IoTDB - Apache Skywalking适配器
Review comment:
Thanks for your suggestion. I will indicate some important design principles in the comments of related codes and delete it from repo in these days. After this project, I will make it as a blog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714439758
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/management/IoTDBUITemplateManagementDAO.java
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.management;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.query.input.DashboardSetting;
+import org.apache.skywalking.oap.server.core.query.type.DashboardConfiguration;
+import org.apache.skywalking.oap.server.core.query.type.TemplateChangeStatus;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.management.UITemplateManagementDAO;
+import org.apache.skywalking.oap.server.library.util.BooleanUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBUITemplateManagementDAO implements UITemplateManagementDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<UITemplate> storageBuilder = new UITemplate.Builder();
+ private static final long UI_TEMPLATE_TIMESTAMP = 1L;
+
+ @Override
+ public List<DashboardConfiguration> getAllTemplates(Boolean includingDisabled) throws IOException {
+ final List<String> indexes = IoTDBTableMetaInfo.get(UITemplate.INDEX_NAME).getIndexes();
Review comment:
Unused
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBHistoryDeleteDAO.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+
+@Slf4j
+public class IoTDBHistoryDeleteDAO implements IHistoryDeleteDAO {
+ private final IoTDBClient client;
+
+ public IoTDBHistoryDeleteDAO(IoTDBClient client) {
+ this.client = client;
+ }
Review comment:
Use `@RequiredArgsConstructor` instead
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBMetricsQueryDAO.java
##########
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.metrics.DataTable;
+import org.apache.skywalking.oap.server.core.query.PointOfTime;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.input.MetricsCondition;
+import org.apache.skywalking.oap.server.core.query.sql.Function;
+import org.apache.skywalking.oap.server.core.query.type.HeatMap;
+import org.apache.skywalking.oap.server.core.query.type.IntValues;
+import org.apache.skywalking.oap.server.core.query.type.KVInt;
+import org.apache.skywalking.oap.server.core.query.type.MetricsValues;
+import org.apache.skywalking.oap.server.core.storage.annotation.ValueColumnMetadata;
+import org.apache.skywalking.oap.server.core.storage.query.IMetricsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+public class IoTDBMetricsQueryDAO implements IMetricsQueryDAO {
+ private final IoTDBClient client;
+
+ public IoTDBMetricsQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public long readMetricsValue(MetricsCondition condition, String valueColumnName, Duration duration) throws IOException {
+ final int defaultValue = ValueColumnMetadata.INSTANCE.getDefaultValue(condition.getName());
+ final Function function = ValueColumnMetadata.INSTANCE.getValueFunction(condition.getName());
+ if (function == Function.Latest) {
+ return readMetricsValues(condition, valueColumnName, duration).getValues().latestValue(defaultValue);
+ }
+
+ StringBuilder query = new StringBuilder();
+ String op;
+ if (function == Function.Avg) {
+ op = "avg";
+ } else {
+ op = "sum";
+ }
+ query.append(String.format("select %s(%s) from ", op, valueColumnName))
+ .append(client.getStorageGroup()).append(IoTDBClient.DOT).append(condition.getName());
+ final String entityId = condition.getEntity().buildId();
+ if (entityId != null) {
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.ENTITY_ID_IDX, entityId);
+ query = client.addQueryIndexValue(condition.getName(), query, indexAndValueMap);
+ } else {
+ query = client.addQueryAsterisk(condition.getName(), query);
+ }
+ query.append(" where ").append(String.format("%s >= %s and %s <= %s",
+ IoTDBClient.TIME, duration.getStartTimestamp(), IoTDBClient.TIME, duration.getEndTimestamp()))
+ .append(" group by level = 3");
+
+ List<Double> results = client.queryWithAgg(condition.getName(), query.toString());
+ if (results.size() > 0) {
+ double result = results.get(0);
+ return (long) result;
+ } else {
+ return defaultValue;
+ }
+ }
+
+ @Override
+ public MetricsValues readMetricsValues(final MetricsCondition condition,
+ final String valueColumnName,
+ final Duration duration) throws IOException {
+ final List<PointOfTime> pointOfTimes = duration.assembleDurationPoints();
+ final List<String> ids = new ArrayList<>(pointOfTimes.size());
+ pointOfTimes.forEach(pointOfTime -> ids.add(pointOfTime.id(condition.getEntity().buildId())));
+
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(valueColumnName).append(" from ");
+ for (String id : ids) {
+ query.append(client.getStorageGroup()).append(IoTDBClient.DOT).append(condition.getName());
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.ID_IDX, id);
+ query = client.addQueryIndexValue(condition.getName(), query, indexAndValueMap);
+ query.append(", ");
+ }
+ String queryString = query.toString();
+ if (ids.size() > 0) {
+ queryString = queryString.substring(0, queryString.lastIndexOf(","));
+ }
+ queryString += IoTDBClient.ALIGN_BY_DEVICE;
+
+ MetricsValues metricsValues = new MetricsValues();
+ // Label is null, because in readMetricsValues, no label parameter.
+ final IntValues intValues = metricsValues.getValues();
+ try {
+ SessionPool sessionPool = client.getSessionPool();
+ if (sessionPool.checkTimeseriesExists(client.getStorageGroup() + IoTDBClient.DOT + condition.getName())) {
+ return metricsValues;
+ }
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement(queryString);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", queryString, wrapper.getColumnNames());
+ }
+
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String id = client.layerName2IndexValue(layerNames[1]);
+ long value = fields.get(1).getLongV();
+
+ KVInt kv = new KVInt();
+ kv.setId(id);
+ kv.setValue(value);
+ intValues.addKVInt(kv);
+ }
+ sessionPool.closeResultSet(wrapper);
Review comment:
You may want to put this in a `finally` block to make sure it's closed anyway
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,476 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.exception.NullFieldException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String GROUP_IDX = "service_group";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) throws IOException {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ public final String getStorageGroup() {
+ return storageGroup;
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize(), config.isRpcCompression());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getTimeseriesList(), request.getTimeseriesTypes(), request.getTimeseriesValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getTimeseriesList());
+ typesList.add(request.getTimeseriesTypes());
+ valuesList.add(request.getTimeseriesValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(modelName);
+ if (!sessionPool.checkTimeseriesExists(devicePath.toString())) {
+ return storageDataList;
+ }
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
+ }
+ if (map.containsKey(IoTDBClient.NODE_TYPE_IDX)) {
+ String nodeType = (String) map.get(IoTDBClient.NODE_TYPE_IDX);
+ map.put(IoTDBClient.NODE_TYPE_IDX, Integer.valueOf(nodeType));
+ }
+ if (modelName.equals(BrowserErrorLogRecord.INDEX_NAME) || modelName.equals(LogRecord.INDEX_NAME)) {
+ map.put(IoTDBClient.TIMESTAMP, map.get("\"" + IoTDBClient.TIMESTAMP + "\""));
+ }
+ for (Map.Entry<String, Object> entry : map.entrySet()) {
+ // remove double quotes
+ String key = entry.getKey();
+ if (key.contains(".")) {
+ map.put(key.substring(1, key.length() - 1), entry.getValue());
+ }
+ }
+
+ storageDataList.add(storageBuilder.storage2Entity(map));
+ }
+ sessionPool.closeResultSet(wrapper);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e.getMessage() + System.lineSeparator() + "SQL Statement: " + querySQL, e);
+ }
+ return storageDataList;
+ }
+
+ /**
+ * query with "string_contains" function
+ *
+ * @param modelName model name
+ * @param querySQL query sql
+ * @param storageBuilder storageBuilder
+ * @return a list of data which has been filtered by "string_contains" function
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> queryWithContains(String modelName, String querySQL,
Review comment:
Unused
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBAggregationQueryDAO.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.input.TopNCondition;
+import org.apache.skywalking.oap.server.core.query.type.KeyValue;
+import org.apache.skywalking.oap.server.core.query.type.SelectedRecord;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+public class IoTDBAggregationQueryDAO implements IAggregationQueryDAO {
+ private IoTDBClient client;
+ private static final Comparator<SelectedRecord> ASCENDING =
+ Comparator.comparingLong(a -> Long.parseLong(a.getValue()));
+ private static final Comparator<SelectedRecord> DESCENDING = (a, b) ->
+ Long.compare(Long.parseLong(b.getValue()), Long.parseLong(a.getValue()));
Review comment:
Unused
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBMetricsDAOTest.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.skywalking.oap.server.core.analysis.DownSampling;
+import org.apache.skywalking.oap.server.core.analysis.FunctionCategory;
+import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic;
+import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.model.ExtraQueryIndex;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBStorageConfig;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+import static org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClientTest.retrieval;
+
+public class IoTDBMetricsDAOTest {
Review comment:
This kind of tests belong to integration test, refer to other existing storage tests about how to test.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,476 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.exception.NullFieldException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String GROUP_IDX = "service_group";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) throws IOException {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ public final String getStorageGroup() {
+ return storageGroup;
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize(), config.isRpcCompression());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getTimeseriesList(), request.getTimeseriesTypes(), request.getTimeseriesValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getTimeseriesList());
+ typesList.add(request.getTimeseriesTypes());
+ valuesList.add(request.getTimeseriesValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(modelName);
+ if (!sessionPool.checkTimeseriesExists(devicePath.toString())) {
+ return storageDataList;
+ }
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
+ }
+ if (map.containsKey(IoTDBClient.NODE_TYPE_IDX)) {
+ String nodeType = (String) map.get(IoTDBClient.NODE_TYPE_IDX);
+ map.put(IoTDBClient.NODE_TYPE_IDX, Integer.valueOf(nodeType));
+ }
+ if (modelName.equals(BrowserErrorLogRecord.INDEX_NAME) || modelName.equals(LogRecord.INDEX_NAME)) {
+ map.put(IoTDBClient.TIMESTAMP, map.get("\"" + IoTDBClient.TIMESTAMP + "\""));
+ }
+ for (Map.Entry<String, Object> entry : map.entrySet()) {
+ // remove double quotes
+ String key = entry.getKey();
+ if (key.contains(".")) {
+ map.put(key.substring(1, key.length() - 1), entry.getValue());
+ }
+ }
+
+ storageDataList.add(storageBuilder.storage2Entity(map));
+ }
+ sessionPool.closeResultSet(wrapper);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e.getMessage() + System.lineSeparator() + "SQL Statement: " + querySQL, e);
+ }
+ return storageDataList;
+ }
+
+ /**
+ * query with "string_contains" function
+ *
+ * @param modelName model name
+ * @param querySQL query sql
+ * @param storageBuilder storageBuilder
+ * @return a list of data which has been filtered by "string_contains" function
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> queryWithContains(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(modelName);
+ if (!sessionPool.checkTimeseriesExists(devicePath.toString())) {
+ return storageDataList;
+ }
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+ List<String> iotDBColumnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+
+ // group columns by device
+ Map<String, List<Integer>> deviceAndTimeseriesMap = new HashMap<>();
+ Map<String, Integer> deviceAndStringContainsMap = new HashMap<>();
+ List<String> timeseriesList = new ArrayList<>(iotDBColumnNames.size());
+ final String stringContains = "string_contains";
+ for (int i = 1; i < iotDBColumnNames.size(); i++) {
+ String iotDBColumnName = iotDBColumnNames.get(i);
+ timeseriesList.add(iotDBColumnName.substring(iotDBColumnName.lastIndexOf(IoTDBClient.DOT) + 1));
+ if (iotDBColumnName.contains(stringContains)) {
+ String device = iotDBColumnName.substring(stringContains.length() + 1, iotDBColumnName.lastIndexOf(IoTDBClient.DOT));
+ deviceAndStringContainsMap.put(device, i);
+ } else {
+ String device = iotDBColumnName.substring(0, iotDBColumnName.lastIndexOf(IoTDBClient.DOT));
+ if (deviceAndTimeseriesMap.containsKey(device)) {
+ deviceAndTimeseriesMap.get(device).add(i);
+ } else {
+ List<Integer> timeseriesIndex = new ArrayList<>(iotDBColumnNames.size());
+ timeseriesIndex.add(i);
+ deviceAndTimeseriesMap.put(device, timeseriesIndex);
+ }
+ }
+ }
+
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new HashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ boolean isStringContains = false;
+ for (Map.Entry<String, Integer> entry : deviceAndStringContainsMap.entrySet()) {
+ String device = entry.getKey();
+ int stringContainsIdx = entry.getValue();
+ if (getStringContainsResult(fields.get(stringContainsIdx - 1))) {
+ isStringContains = true;
+ String[] layerNames = device.split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ List<Integer> timeseriesIndexes = deviceAndTimeseriesMap.get(device);
+ timeseriesIndexes.forEach(timeseriesIndex -> {
+ String timeseries = timeseriesList.get(timeseriesIndex - 1);
+ Field field = fields.get(timeseriesIndex - 1);
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(timeseries, field.getStringValue());
+ } else {
+ map.put(timeseries, field.getObjectValue(field.getDataType()));
+ }
+ });
+ break;
+ }
+ }
+ if (map.containsKey(IoTDBClient.NODE_TYPE_IDX)) {
+ String nodeType = (String) map.get(IoTDBClient.NODE_TYPE_IDX);
+ map.put(IoTDBClient.NODE_TYPE_IDX, Integer.valueOf(nodeType));
+ }
+
+ if (isStringContains) {
+ storageDataList.add(storageBuilder.storage2Entity(map));
+ }
+ }
+ sessionPool.closeResultSet(wrapper);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e.getMessage() + System.lineSeparator() + "SQL Statement: " + querySQL, e);
+ }
+ return storageDataList;
+ }
+
+ public List<Long> queryWithSelect(String modelName, String querySQL) throws IOException {
Review comment:
Unused
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBAggregationQueryDAO.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.input.TopNCondition;
+import org.apache.skywalking.oap.server.core.query.type.KeyValue;
+import org.apache.skywalking.oap.server.core.query.type.SelectedRecord;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+public class IoTDBAggregationQueryDAO implements IAggregationQueryDAO {
+ private IoTDBClient client;
Review comment:
can be final
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBBatchDAO.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.storage.IBatchDAO;
+import org.apache.skywalking.oap.server.library.client.request.InsertRequest;
+import org.apache.skywalking.oap.server.library.client.request.PrepareRequest;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Slf4j
+public class IoTDBBatchDAO implements IBatchDAO {
+ private final IoTDBClient client;
+
+ public IoTDBBatchDAO(IoTDBClient client) {
+ this.client = client;
+ }
Review comment:
```suggestion
@RequiredArgsConstructor
public class IoTDBBatchDAO implements IBatchDAO {
private final IoTDBClient client;
```
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBMetricsDAOTest.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.skywalking.oap.server.core.analysis.DownSampling;
+import org.apache.skywalking.oap.server.core.analysis.FunctionCategory;
+import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic;
+import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.model.ExtraQueryIndex;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBStorageConfig;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+import static org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClientTest.retrieval;
+
+public class IoTDBMetricsDAOTest {
+ private IoTDBMetricsDAO metricsDAO;
+
+ @Before
+ public void setUp() throws Exception {
+ IoTDBStorageConfig config = new IoTDBStorageConfig();
+ config.setHost("127.0.0.1");
+ config.setRpcPort(6667);
+ config.setUsername("root");
+ config.setPassword("root");
+ config.setStorageGroup("root.skywalking");
+ config.setSessionPoolSize(3);
+ config.setFetchTaskLogMaxSize(1000);
+
+ IoTDBClient client = new IoTDBClient(config);
+ client.connect();
+
+ StorageHashMapBuilder<? extends Metrics> storageBuilder = new InstanceTraffic.Builder();
+ metricsDAO = new IoTDBMetricsDAO(client, (StorageHashMapBuilder<Metrics>) storageBuilder);
+
+ int scopeId = 1;
+ boolean record = false;
+ boolean superDataset = false;
+ boolean timeRelativeID = true;
+ List<ModelColumn> modelColumns = new ArrayList<>();
+ List<ExtraQueryIndex> extraQueryIndices = new ArrayList<>();
+ retrieval(InstanceTraffic.class, InstanceTraffic.INDEX_NAME, modelColumns, extraQueryIndices, scopeId);
+ Model instanceTrafficModel = new Model(
+ InstanceTraffic.INDEX_NAME, modelColumns, extraQueryIndices, scopeId, DownSampling.Second, record,
+ superDataset, FunctionCategory.uniqueFunctionName(InstanceTraffic.class), timeRelativeID);
+ IoTDBTableMetaInfo.addModel(instanceTrafficModel);
+
+ StorageHashMapBuilder<InstanceTraffic> instanceTrafficBuilder = new InstanceTraffic.Builder();
+ Map<String, Object> instanceTrafficMap = new HashMap<>();
+ instanceTrafficMap.put(InstanceTraffic.SERVICE_ID, "service_id_1");
+ instanceTrafficMap.put(InstanceTraffic.NAME, "name_1");
+ instanceTrafficMap.put(InstanceTraffic.LAST_PING_TIME_BUCKET, 1L);
+ instanceTrafficMap.put(InstanceTraffic.PROPERTIES, "{test: 1}");
+ instanceTrafficMap.put(InstanceTraffic.ENTITY_ID, "entity_id_1");
+ instanceTrafficMap.put(InstanceTraffic.TIME_BUCKET, 4L);
+ InstanceTraffic instanceTraffic = instanceTrafficBuilder.storage2Entity(instanceTrafficMap);
+ IoTDBInsertRequest request = new IoTDBInsertRequest(InstanceTraffic.INDEX_NAME, 4L, instanceTraffic, instanceTrafficBuilder);
+ client.write(request);
+
+ instanceTrafficMap.put(InstanceTraffic.SERVICE_ID, "service_id_2");
+ instanceTrafficMap.put(InstanceTraffic.NAME, "name_2");
+ instanceTrafficMap.put(InstanceTraffic.LAST_PING_TIME_BUCKET, 2L);
+ instanceTrafficMap.put(InstanceTraffic.PROPERTIES, "{test: 2}");
+ instanceTrafficMap.put(InstanceTraffic.ENTITY_ID, "entity_id_2");
+ instanceTrafficMap.put(InstanceTraffic.TIME_BUCKET, 4L);
+ instanceTraffic = instanceTrafficBuilder.storage2Entity(instanceTrafficMap);
+ request = new IoTDBInsertRequest(InstanceTraffic.INDEX_NAME, 2L, instanceTraffic, instanceTrafficBuilder);
+ client.write(request);
+ }
+
+ @Test
+ public void multiGet() throws IOException {
+ List<Metrics> metrics = new ArrayList<>();
+ InstanceTraffic instanceTraffic1 = new InstanceTraffic();
+ instanceTraffic1.setName("name_1");
+ instanceTraffic1.setServiceId("service_id_1");
+ metrics.add(instanceTraffic1);
+ InstanceTraffic instanceTraffic2 = new InstanceTraffic();
+ instanceTraffic2.setName("name_2");
+ instanceTraffic2.setServiceId("service_id_2");
+ metrics.add(instanceTraffic2);
+
+ List<Metrics> newMetrics = metricsDAO.multiGet(IoTDBTableMetaInfo.get(InstanceTraffic.INDEX_NAME).getModel(), metrics);
+ newMetrics.forEach(metrics1 -> {
+ String serviceId = ((InstanceTraffic) metrics1).getServiceId();
+ String name = ((InstanceTraffic) metrics1).getName();
+ assert serviceId.equals("service_id_1") || serviceId.equals("service_id_2");
+ assert name.equals("name_1") || name.equals("name_2");
Review comment:
Do not use `assert `, use `org.assertj.core.api.Assertions#assertThat` instead
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBTraceQueryDAO.java
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.skywalking.apm.util.StringUtil;
+import org.apache.skywalking.oap.server.core.analysis.IDManager;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
+import org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord;
+import org.apache.skywalking.oap.server.core.query.type.BasicTrace;
+import org.apache.skywalking.oap.server.core.query.type.QueryOrder;
+import org.apache.skywalking.oap.server.core.query.type.Span;
+import org.apache.skywalking.oap.server.core.query.type.TraceBrief;
+import org.apache.skywalking.oap.server.core.query.type.TraceState;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
+import org.apache.skywalking.oap.server.library.util.BooleanUtils;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+public class IoTDBTraceQueryDAO implements ITraceQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<SegmentRecord> storageBuilder = new SegmentRecord.Builder();
+
+ public IoTDBTraceQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public TraceBrief queryBasicTraces(long startSecondTB, long endSecondTB, long minDuration, long maxDuration,
+ String serviceId, String serviceInstanceId,
+ String endpointId, String traceId, int limit, int from,
+ TraceState traceState, QueryOrder queryOrder, List<Tag> tags)
+ throws IOException {
+ StringBuilder query = new StringBuilder();
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(SegmentRecord.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ if (StringUtil.isNotEmpty(serviceId)) {
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ }
+ if (!Strings.isNullOrEmpty(traceId)) {
+ indexAndValueMap.put(IoTDBClient.TRACE_ID_IDX, traceId);
+ }
+ query = client.addQueryIndexValue(SegmentRecord.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where 1=1");
+ if (startSecondTB != 0 && endSecondTB != 0) {
+ query.append(" and ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startSecondTB));
+ query.append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endSecondTB));
+ }
+ if (minDuration != 0) {
+ query.append(" and ").append(SegmentRecord.LATENCY).append(" >= ").append(minDuration);
+ }
+ if (maxDuration != 0) {
+ query.append(" and ").append(SegmentRecord.LATENCY).append(" <= ").append(maxDuration);
+ }
+ if (StringUtil.isNotEmpty(serviceInstanceId)) {
+ query.append(" and ").append(SegmentRecord.SERVICE_INSTANCE_ID).append(" = \"").append(serviceInstanceId).append("\"");
+ }
+ if (!Strings.isNullOrEmpty(endpointId)) {
+ query.append(" and ").append(SegmentRecord.ENDPOINT_ID).append(" = \"").append(endpointId).append("\"");
+ }
+ if (CollectionUtils.isNotEmpty(tags)) {
+ for (final Tag tag : tags) {
+ query.append(" and ").append(tag.getKey()).append(" = \"").append(tag.getValue()).append("\"");
+ }
+ }
+ switch (traceState) {
+ case ERROR:
+ query.append(" and ").append(SegmentRecord.IS_ERROR).append(" = ").append(BooleanUtils.TRUE);
+ break;
+ case SUCCESS:
+ query.append(" and ").append(SegmentRecord.IS_ERROR).append(" = ").append(BooleanUtils.FALSE);
+ break;
+ }
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString().replace("1=1 and ", "");
+ queryString = queryString + IoTDBClient.ALIGN_BY_DEVICE;
+
+ TraceBrief traceBrief = new TraceBrief();
+ List<? super StorageData> storageDataList = client.filterQuery(SegmentRecord.INDEX_NAME, queryString, storageBuilder);
+ int limitCount = 0;
+ for (int i = 0; i < storageDataList.size(); i++) {
+ if (i >= from && limitCount < limit) {
+ limitCount++;
+ SegmentRecord segmentRecord = (SegmentRecord) storageDataList.get(i);
+ BasicTrace basicTrace = new BasicTrace();
+ basicTrace.setSegmentId(segmentRecord.getSegmentId());
+ basicTrace.setStart(String.valueOf(segmentRecord.getStartTime()));
+ basicTrace.getEndpointNames().add(IDManager.EndpointID.analysisId(segmentRecord.getEndpointId()).getEndpointName());
+ basicTrace.setDuration(segmentRecord.getLatency());
+ basicTrace.setError(BooleanUtils.valueToBoolean(segmentRecord.getIsError()));
+ basicTrace.getTraceIds().add(segmentRecord.getTraceId());
+ traceBrief.getTraces().add(basicTrace);
+ }
+ }
+ traceBrief.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ switch (queryOrder) {
+ case BY_START_TIME:
+ traceBrief.getTraces().sort((BasicTrace b1, BasicTrace b2) ->
+ Long.compare(Long.parseLong(b2.getStart()), Long.parseLong(b1.getStart())));
+ break;
+ case BY_DURATION:
+ traceBrief.getTraces().sort((BasicTrace b1, BasicTrace b2) ->
+ Integer.compare(b2.getDuration(), b1.getDuration()));
+ break;
+ }
+ return traceBrief;
+ }
+
+ @Override
+ public List<SegmentRecord> queryByTraceId(String traceId) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(SegmentRecord.INDEX_NAME);
Review comment:
Consider adding an index formatter instead of concatenating this part over and over, like https://github.com/apache/skywalking/blob/4a344b01e9ad103b1cfda3e56d774dd4778c77de/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/StorageModuleElasticsearchProvider.java#L225-L232
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,53 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <iotdb-session.version>0.12.2</iotdb-session.version>
Review comment:
Unused
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,476 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.exception.NullFieldException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String GROUP_IDX = "service_group";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) throws IOException {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ public final String getStorageGroup() {
+ return storageGroup;
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize(), config.isRpcCompression());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getTimeseriesList(), request.getTimeseriesTypes(), request.getTimeseriesValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getTimeseriesList());
+ typesList.add(request.getTimeseriesTypes());
+ valuesList.add(request.getTimeseriesValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(modelName);
+ if (!sessionPool.checkTimeseriesExists(devicePath.toString())) {
+ return storageDataList;
+ }
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
+ }
+ if (map.containsKey(IoTDBClient.NODE_TYPE_IDX)) {
+ String nodeType = (String) map.get(IoTDBClient.NODE_TYPE_IDX);
+ map.put(IoTDBClient.NODE_TYPE_IDX, Integer.valueOf(nodeType));
+ }
+ if (modelName.equals(BrowserErrorLogRecord.INDEX_NAME) || modelName.equals(LogRecord.INDEX_NAME)) {
+ map.put(IoTDBClient.TIMESTAMP, map.get("\"" + IoTDBClient.TIMESTAMP + "\""));
+ }
+ for (Map.Entry<String, Object> entry : map.entrySet()) {
+ // remove double quotes
+ String key = entry.getKey();
+ if (key.contains(".")) {
+ map.put(key.substring(1, key.length() - 1), entry.getValue());
+ }
+ }
+
+ storageDataList.add(storageBuilder.storage2Entity(map));
+ }
+ sessionPool.closeResultSet(wrapper);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e.getMessage() + System.lineSeparator() + "SQL Statement: " + querySQL, e);
+ }
+ return storageDataList;
+ }
+
+ /**
+ * query with "string_contains" function
+ *
+ * @param modelName model name
+ * @param querySQL query sql
+ * @param storageBuilder storageBuilder
+ * @return a list of data which has been filtered by "string_contains" function
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> queryWithContains(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(modelName);
+ if (!sessionPool.checkTimeseriesExists(devicePath.toString())) {
+ return storageDataList;
+ }
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+ List<String> iotDBColumnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+
+ // group columns by device
+ Map<String, List<Integer>> deviceAndTimeseriesMap = new HashMap<>();
+ Map<String, Integer> deviceAndStringContainsMap = new HashMap<>();
+ List<String> timeseriesList = new ArrayList<>(iotDBColumnNames.size());
+ final String stringContains = "string_contains";
+ for (int i = 1; i < iotDBColumnNames.size(); i++) {
+ String iotDBColumnName = iotDBColumnNames.get(i);
+ timeseriesList.add(iotDBColumnName.substring(iotDBColumnName.lastIndexOf(IoTDBClient.DOT) + 1));
+ if (iotDBColumnName.contains(stringContains)) {
+ String device = iotDBColumnName.substring(stringContains.length() + 1, iotDBColumnName.lastIndexOf(IoTDBClient.DOT));
+ deviceAndStringContainsMap.put(device, i);
+ } else {
+ String device = iotDBColumnName.substring(0, iotDBColumnName.lastIndexOf(IoTDBClient.DOT));
+ if (deviceAndTimeseriesMap.containsKey(device)) {
+ deviceAndTimeseriesMap.get(device).add(i);
+ } else {
+ List<Integer> timeseriesIndex = new ArrayList<>(iotDBColumnNames.size());
+ timeseriesIndex.add(i);
+ deviceAndTimeseriesMap.put(device, timeseriesIndex);
+ }
+ }
+ }
+
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new HashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ boolean isStringContains = false;
+ for (Map.Entry<String, Integer> entry : deviceAndStringContainsMap.entrySet()) {
+ String device = entry.getKey();
+ int stringContainsIdx = entry.getValue();
+ if (getStringContainsResult(fields.get(stringContainsIdx - 1))) {
+ isStringContains = true;
+ String[] layerNames = device.split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ List<Integer> timeseriesIndexes = deviceAndTimeseriesMap.get(device);
+ timeseriesIndexes.forEach(timeseriesIndex -> {
+ String timeseries = timeseriesList.get(timeseriesIndex - 1);
+ Field field = fields.get(timeseriesIndex - 1);
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(timeseries, field.getStringValue());
+ } else {
+ map.put(timeseries, field.getObjectValue(field.getDataType()));
+ }
+ });
+ break;
+ }
+ }
+ if (map.containsKey(IoTDBClient.NODE_TYPE_IDX)) {
+ String nodeType = (String) map.get(IoTDBClient.NODE_TYPE_IDX);
+ map.put(IoTDBClient.NODE_TYPE_IDX, Integer.valueOf(nodeType));
+ }
+
+ if (isStringContains) {
+ storageDataList.add(storageBuilder.storage2Entity(map));
+ }
+ }
+ sessionPool.closeResultSet(wrapper);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e.getMessage() + System.lineSeparator() + "SQL Statement: " + querySQL, e);
+ }
+ return storageDataList;
+ }
+
+ public List<Long> queryWithSelect(String modelName, String querySQL) throws IOException {
+ // TODO refactor select query, https://github.com/apache/iotdb/discussions/3888
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(modelName);
+ SessionDataSetWrapper wrapper = null;
+ try {
+ if (!sessionPool.checkTimeseriesExists(devicePath.toString())) {
+ return null;
+ }
+ wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+ List<Long> longList = new ArrayList<>();
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ longList.add(rowRecord.getFields().get(1).getLongV());
+ }
+ healthChecker.health();
+ return longList;
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e.getMessage() + System.lineSeparator() + "SQL Statement: " + querySQL, e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ }
+
+ /**
+ * Query with aggregation function: count, sum, avg, last_value, first_value, min_time, max_time, min_value, max_value
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which should contain aggregation function
+ * @return the result of aggregation function
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<Double> queryWithAgg(String modelName, String querySQL) throws IOException {
+ try {
+ List<Double> results = new ArrayList<>();
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(modelName);
+ if (!sessionPool.checkTimeseriesExists(devicePath.toString())) {
+ return results;
+ }
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ if (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ for (Field field : fields) {
+ String stringValue = field.getStringValue();
+ if (!stringValue.equals("null")) {
+ results.add(Double.parseDouble(stringValue));
+ }
+ }
+ }
+ healthChecker.health();
+ sessionPool.closeResultSet(wrapper);
+ return results;
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e.getMessage() + System.lineSeparator() + "SQL Statement: " + querySQL, e);
+ }
+ }
+
+ /**
+ * Delete data <= deleteTime in one timeseries
+ *
+ * @param device device name
+ * @param deleteTime deleteTime
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void deleteData(String device, long deleteTime) throws IOException {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(device);
+ try {
+ sessionPool.deleteData(devicePath.toString(), deleteTime);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ public String indexValue2LayerName(String indexValue) {
+ return "\"" + indexValue + "\"";
+ }
+
+ public String layerName2IndexValue(String layerName) {
+ return layerName.substring(0, layerName.length() - 1);
+ }
+
+ public StringBuilder addQueryIndexValue(String modelName, StringBuilder query, Map<String, String> indexAndValueMap) {
+ List<String> indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexes.forEach(index -> {
+ if (indexAndValueMap.containsKey(index)) {
+ query.append(IoTDBClient.DOT).append(indexValue2LayerName(indexAndValueMap.get(index)));
+ } else {
+ query.append(IoTDBClient.DOT).append("*");
+ }
+ });
+ return query;
+ }
+
+ public StringBuilder addQueryAsterisk(String modelName, StringBuilder query) {
+ List<String> indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexes.forEach(index -> query.append(IoTDBClient.DOT).append("*"));
+ return query;
+ }
+
+ private boolean getStringContainsResult(Field field) {
Review comment:
Unused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r742501514
##########
File path: oap-server-bom/pom.xml
##########
@@ -77,6 +77,7 @@
<awaitility.version>3.0.0</awaitility.version>
<httpcore.version>4.4.13</httpcore.version>
<commons-compress.version>1.21</commons-compress.version>
+ <iotdb-session.version>0.13.0-SNAPSHOT</iotdb-session.version>
Review comment:
FYI, we need to fix this before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715281992
##########
File path: dist-material/release-docs/LICENSE
##########
@@ -327,6 +327,7 @@ The text of each license is the standard Apache 2.0 license.
Armeria 1.11.0, http://github.com/line/armeria, Apache 2.0
Brotli4j 1.6.0, https://github.com/hyperxpro/Brotli4j, Apache 2.0
micrometer 1.7.3, https://github.com/micrometer-metrics/micrometer, Apache 2.0
+ iotdb-session 0.12.2, https://github.com/apache/iotdb, Apache 2.0
Review comment:
I've modified `known-oap-backend-dependencies.txt` before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r744257052
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,17 +61,26 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.13.0-SNAPSHOT.jar
+iotdb-thrift-0.13.0-SNAPSHOT.jar
+istack-commons-runtime-4.0.0.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
jackson-databind-2.12.2.jar
jackson-datatype-guava-2.9.10.jar
jackson-datatype-jdk8-2.9.10.jar
jackson-module-afterburner-2.12.2.jar
jackson-module-kotlin-2.8.8.jar
+jakarta.activation-2.0.0.jar
+jakarta.xml.bind-api-3.0.0.jar
Review comment:
I also don't find any related jars in the dependency tree of new module storage-iotdb-plugin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753768185
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
@kezhenxu94 I can see this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753769791
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
OK, sorry @LIU-WEI-git , it does indeed include an old version of `lz4`, please add the `exclusion` back and move the new `lz4-java` to the `bom` module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755802092
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBTopologyQueryDAO.java
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.endpoint.EndpointRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.query.type.Call;
+import org.apache.skywalking.oap.server.core.source.DetectPoint;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBTopologyQueryDAO implements ITopologyQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtServerSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationServerSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtClientSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationClientSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadEndpointRelation(long startTB, long endTB, String destEndpointId) throws IOException {
+ List<Call.CallDetail> calls = loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, false);
+ calls.addAll(loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, true));
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ List<String> serviceIds, DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ if (serviceIds.size() > 0) {
+ query.append(" and (");
+ for (int i = 0; i < serviceIds.size(); i++) {
+ query.append(sourceCName).append(" = \"").append(serviceIds.get(i))
+ .append("\" or ")
+ .append(destCName).append(" = \"").append(serviceIds.get(i)).append("\"");
+ if (i != serviceIds.size() - 1) {
+ query.append(" or ");
+ }
+ }
+ query.append(")");
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromServiceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceInstanceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String descCName,
+ String sourceServiceId, String destServiceId,
+ DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceInstanceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ((").append(sourceCName).append(" = \"").append(sourceServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(destServiceId).append("\"")
+ .append(") or (").append(sourceCName).append(" = \"").append(destServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(sourceServiceId).append(" \"))")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromInstanceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadEndpointFromSide(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ String id, boolean isSourceId) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ").append(isSourceId ? sourceCName : destCName).append(" = \"").append(id).append("\"")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ // TODO check DetectPoint
+ call.buildFromEndpointRelation(entityId, DetectPoint.SERVER);
Review comment:
The difference is in `loadEndpointRelation` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755799051
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBTopologyQueryDAO.java
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.endpoint.EndpointRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.query.type.Call;
+import org.apache.skywalking.oap.server.core.source.DetectPoint;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBTopologyQueryDAO implements ITopologyQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtServerSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationServerSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtClientSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationClientSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadEndpointRelation(long startTB, long endTB, String destEndpointId) throws IOException {
+ List<Call.CallDetail> calls = loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, false);
+ calls.addAll(loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, true));
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ List<String> serviceIds, DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ if (serviceIds.size() > 0) {
+ query.append(" and (");
+ for (int i = 0; i < serviceIds.size(); i++) {
+ query.append(sourceCName).append(" = \"").append(serviceIds.get(i))
+ .append("\" or ")
+ .append(destCName).append(" = \"").append(serviceIds.get(i)).append("\"");
+ if (i != serviceIds.size() - 1) {
+ query.append(" or ");
+ }
+ }
+ query.append(")");
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromServiceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceInstanceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String descCName,
+ String sourceServiceId, String destServiceId,
+ DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceInstanceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ((").append(sourceCName).append(" = \"").append(sourceServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(destServiceId).append("\"")
+ .append(") or (").append(sourceCName).append(" = \"").append(destServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(sourceServiceId).append(" \"))")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromInstanceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadEndpointFromSide(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ String id, boolean isSourceId) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ").append(isSourceId ? sourceCName : destCName).append(" = \"").append(id).append("\"")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ // TODO check DetectPoint
+ call.buildFromEndpointRelation(entityId, DetectPoint.SERVER);
Review comment:
> This is client side detecting, what do you mean exactly?
No, it's not this method. In line 291 in `TopologyQueryEsDAO` and line 270 in `H2TopologyQueryDAO`, the second parameter of `buildFromEndpointRelation` always is `DetectPoint.SERVER`. But in line 305 of `TopologyQuery`, it can be `DetectPoint.SERVER` or `DetectPoint.CLIENT`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715282661
##########
File path: dist-material/release-docs/LICENSE
##########
@@ -327,6 +327,7 @@ The text of each license is the standard Apache 2.0 license.
Armeria 1.11.0, http://github.com/line/armeria, Apache 2.0
Brotli4j 1.6.0, https://github.com/hyperxpro/Brotli4j, Apache 2.0
micrometer 1.7.3, https://github.com/micrometer-metrics/micrometer, Apache 2.0
+ iotdb-session 0.12.2, https://github.com/apache/iotdb, Apache 2.0
Review comment:
You just added one line, we check for all real dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r742501838
##########
File path: test/e2e-v2/cases/alarm/iotdb/docker-compose.yml
##########
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ iotdb:
+# image: apache/iotdb:0.12.2-node
+ image: timcross0/iotdb
Review comment:
FYI, we need to fix this before merging, including all CI e2e related images.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r757202725
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String GROUP_IDX = "service_group";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getMeasurements(), request.getMeasurementTypes(), request.getMeasurementValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ if (log.isDebugEnabled()) {
+ for (IoTDBInsertRequest request : requestList) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+ }
+
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getMeasurements());
+ typesList.add(request.getMeasurementTypes());
+ valuesList.add(request.getMeasurementValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ SessionDataSetWrapper wrapper = null;
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
Review comment:
You may need to check whether the `field` is null or not, it seems `field.getStringValue()` returns a string `"null"` if `field` is null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] dmsolr commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
dmsolr commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755678217
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBBrowserLogQueryDAO.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.browser.source.BrowserErrorCategory;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLog;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLogs;
+import org.apache.skywalking.oap.server.core.query.type.ErrorCategory;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.library.util.StringUtil;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBBrowserLogQueryDAO implements IBrowserLogQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<BrowserErrorLogRecord> storageBuilder = new BrowserErrorLogRecord.Builder();
+
+ @Override
+ public BrowserErrorLogs queryBrowserErrorLogs(String serviceId, String serviceVersionId, String pagePathId,
+ BrowserErrorCategory category, long startSecondTB,
+ long endSecondTB, int limit, int from) throws IOException {
+ StringBuilder query = new StringBuilder();
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query.append("select * from ");
+ query = client.addModelPath(query, BrowserErrorLogRecord.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ if (StringUtil.isNotEmpty(serviceId)) {
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ }
+ query = client.addQueryIndexValue(BrowserErrorLogRecord.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where 1=1");
+ if (startSecondTB != 0 && endSecondTB != 0) {
+ query.append(" and ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startSecondTB));
+ query.append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endSecondTB));
+ }
+ if (StringUtil.isNotEmpty(serviceVersionId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.SERVICE_VERSION_ID).append(" = \"").append(serviceVersionId).append("\"");
+ }
+ if (StringUtil.isNotEmpty(pagePathId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.PAGE_PATH_ID).append(" = \"").append(pagePathId).append("\"");
+ }
+ if (Objects.nonNull(category)) {
+ query.append(" and ").append(BrowserErrorLogRecord.ERROR_CATEGORY).append(" = ").append(category.getValue());
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString();
+ queryString = queryString.replace("1=1 and ", "");
Review comment:
Is there a case that does not contain any condition? If so, the result will be `select * from tbl where 1=1`
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBEventQueryDAO.java
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.query.PaginationUtils;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.type.event.EventQueryCondition;
+import org.apache.skywalking.oap.server.core.query.type.event.EventType;
+import org.apache.skywalking.oap.server.core.query.type.event.Events;
+import org.apache.skywalking.oap.server.core.query.type.event.Source;
+import org.apache.skywalking.oap.server.core.source.Event;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBEventQueryDAO implements IEventQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<Event> storageBuilder = new Event.Builder();
+
+ @Override
+ public Events queryEvents(EventQueryCondition condition) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(condition, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = page.getFrom(); i < storageDataList.size(); i++) {
+ if (limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ events.getEvents().add(parseEvent(event));
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
+ (org.apache.skywalking.oap.server.core.query.type.event.Event e1,
+ org.apache.skywalking.oap.server.core.query.type.event.Event e2)
+ -> Long.compare(e2.getStartTime(), e1.getStartTime()));
+ } else {
+ events.getEvents().sort(
+ Comparator.comparingLong(org.apache.skywalking.oap.server.core.query.type.event.Event::getStartTime));
+ }
+ return events;
+ }
+
+ @Override
+ public Events queryEvents(List<EventQueryCondition> conditionList) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(conditionList, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ EventQueryCondition condition = conditionList.get(0);
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = page.getFrom(); i < storageDataList.size(); i++) {
+ if (limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ events.getEvents().add(parseEvent(event));
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
+ (org.apache.skywalking.oap.server.core.query.type.event.Event e1,
+ org.apache.skywalking.oap.server.core.query.type.event.Event e2)
+ -> Long.compare(e2.getStartTime(), e1.getStartTime()));
+ } else {
+ events.getEvents().sort(
+ Comparator.comparingLong(org.apache.skywalking.oap.server.core.query.type.event.Event::getStartTime));
+ }
+ return events;
+ }
+
+ private StringBuilder whereSQL(final EventQueryCondition condition, StringBuilder query) {
+ query.append(" where 1=1");
+ if (!Strings.isNullOrEmpty(condition.getUuid())) {
+ query.append(" and ").append(Event.UUID).append(" = \"").append(condition.getUuid()).append("\"");
+ }
+ final Source source = condition.getSource();
+ if (source != null) {
+ if (!Strings.isNullOrEmpty(source.getService())) {
+ query.append(" and ").append(Event.SERVICE).append(" = \"").append(source.getService()).append("\"");
+ }
+ if (!Strings.isNullOrEmpty(source.getServiceInstance())) {
+ query.append(" and ").append(Event.SERVICE_INSTANCE).append(" = \"").append(source.getServiceInstance()).append("\"");
+ }
+ if (!Strings.isNullOrEmpty(source.getEndpoint())) {
+ query.append(" and ").append(Event.ENDPOINT).append(" = \"").append(source.getEndpoint()).append("\"");
+ }
+ }
+ if (!Strings.isNullOrEmpty(condition.getName())) {
+ query.append(" and ").append(Event.NAME).append(" = \"").append(condition.getName()).append("\"");
+ }
+ if (condition.getType() != null) {
+ query.append(" and ").append(Event.TYPE).append(" = \"").append(condition.getType().name()).append("\"");
+ }
+ final Duration time = condition.getTime();
+ if (time != null) {
+ if (time.getStartTimestamp() > 0) {
+ query.append(" and ").append(Event.START_TIME).append(" > ").append(time.getStartTimestamp());
+ }
+ if (time.getEndTimestamp() > 0) {
+ query.append(" and ").append(Event.END_TIME).append(" < ").append(time.getEndTimestamp());
+ }
+ }
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString();
+ queryString = queryString.replace("1=1 and ", "");
+ return new StringBuilder(queryString);
+ }
Review comment:
A suggestion,
we can be the where conditions in a new StringBuilder to avoid operating string frequently. like following:
```java
StringBuilder where = new StringBuilder();
if (!Strigs.isNullOrEmpty(condition.getUuid())) {
where.append(Event.UUID).append(" = \"").append(condition.getUuid()).append("\"").append(" and ");
}
if (where.length() > 0) {
int length = where.length();
where.delete(length - 4, length);
query.append(" where ").append(where);
}
return query;
```
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBEventQueryDAO.java
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.query.PaginationUtils;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.type.event.EventQueryCondition;
+import org.apache.skywalking.oap.server.core.query.type.event.EventType;
+import org.apache.skywalking.oap.server.core.query.type.event.Events;
+import org.apache.skywalking.oap.server.core.query.type.event.Source;
+import org.apache.skywalking.oap.server.core.source.Event;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBEventQueryDAO implements IEventQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<Event> storageBuilder = new Event.Builder();
+
+ @Override
+ public Events queryEvents(EventQueryCondition condition) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(condition, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = page.getFrom(); i < storageDataList.size(); i++) {
+ if (limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ events.getEvents().add(parseEvent(event));
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
+ (org.apache.skywalking.oap.server.core.query.type.event.Event e1,
+ org.apache.skywalking.oap.server.core.query.type.event.Event e2)
+ -> Long.compare(e2.getStartTime(), e1.getStartTime()));
+ } else {
+ events.getEvents().sort(
+ Comparator.comparingLong(org.apache.skywalking.oap.server.core.query.type.event.Event::getStartTime));
+ }
+ return events;
+ }
+
+ @Override
+ public Events queryEvents(List<EventQueryCondition> conditionList) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(conditionList, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ EventQueryCondition condition = conditionList.get(0);
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = page.getFrom(); i < storageDataList.size(); i++) {
+ if (limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ events.getEvents().add(parseEvent(event));
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
+ (org.apache.skywalking.oap.server.core.query.type.event.Event e1,
+ org.apache.skywalking.oap.server.core.query.type.event.Event e2)
+ -> Long.compare(e2.getStartTime(), e1.getStartTime()));
+ } else {
+ events.getEvents().sort(
+ Comparator.comparingLong(org.apache.skywalking.oap.server.core.query.type.event.Event::getStartTime));
+ }
+ return events;
+ }
+
+ private StringBuilder whereSQL(final EventQueryCondition condition, StringBuilder query) {
+ query.append(" where 1=1");
+ if (!Strings.isNullOrEmpty(condition.getUuid())) {
+ query.append(" and ").append(Event.UUID).append(" = \"").append(condition.getUuid()).append("\"");
+ }
+ final Source source = condition.getSource();
+ if (source != null) {
+ if (!Strings.isNullOrEmpty(source.getService())) {
+ query.append(" and ").append(Event.SERVICE).append(" = \"").append(source.getService()).append("\"");
+ }
+ if (!Strings.isNullOrEmpty(source.getServiceInstance())) {
+ query.append(" and ").append(Event.SERVICE_INSTANCE).append(" = \"").append(source.getServiceInstance()).append("\"");
+ }
+ if (!Strings.isNullOrEmpty(source.getEndpoint())) {
+ query.append(" and ").append(Event.ENDPOINT).append(" = \"").append(source.getEndpoint()).append("\"");
+ }
+ }
+ if (!Strings.isNullOrEmpty(condition.getName())) {
+ query.append(" and ").append(Event.NAME).append(" = \"").append(condition.getName()).append("\"");
+ }
+ if (condition.getType() != null) {
+ query.append(" and ").append(Event.TYPE).append(" = \"").append(condition.getType().name()).append("\"");
+ }
+ final Duration time = condition.getTime();
+ if (time != null) {
+ if (time.getStartTimestamp() > 0) {
+ query.append(" and ").append(Event.START_TIME).append(" > ").append(time.getStartTimestamp());
+ }
+ if (time.getEndTimestamp() > 0) {
+ query.append(" and ").append(Event.END_TIME).append(" < ").append(time.getEndTimestamp());
+ }
+ }
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString();
+ queryString = queryString.replace("1=1 and ", "");
+ return new StringBuilder(queryString);
+ }
Review comment:
the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-979654586
Others LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r757443544
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBInsertRequest.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+import org.apache.skywalking.oap.server.library.client.request.InsertRequest;
+import org.apache.skywalking.oap.server.library.client.request.UpdateRequest;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBIndexes;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+
+@Getter
+@Setter
+@ToString
+@Slf4j
+public class IoTDBInsertRequest implements InsertRequest, UpdateRequest {
+ private String modelName;
+ private long time;
+ private List<String> indexes;
+ private List<String> indexValues;
+ private List<String> measurements;
+ private List<TSDataType> measurementTypes;
+ private List<Object> measurementValues;
+
+ public <T extends StorageData> IoTDBInsertRequest(String modelName, long time, T storageData,
+ StorageHashMapBuilder<T> storageBuilder) {
+ this.modelName = modelName;
+ this.time = time;
+ indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexValues = new ArrayList<>(indexes.size());
+ Map<String, Object> storageMap = storageBuilder.entity2Storage(storageData);
+
+ indexes.forEach(index -> {
+ if (index.equals(IoTDBIndexes.ID_IDX)) {
+ indexValues.add(storageData.id());
+ } else if (storageMap.containsKey(index)) {
+ // avoid `service_group` be "null" when inserting
+ if (index.equals(IoTDBIndexes.GROUP_IDX) && storageMap.get(index) == null) {
Review comment:
Let’s apply this to other fields. I don’t think we want “null” string for null value in all other fields. @wu-sheng correct me if I’m wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753775628
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBHistoryDeleteDAO.java
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBHistoryDeleteDAO implements IHistoryDeleteDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public void deleteHistory(Model model, String timeBucketColumnName, int ttl) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("TTL execution log, model: {}, TTL: {}", model.getName(), ttl);
+ }
Review comment:
Do we really need logs like this? Even we need, I think we should move this to general DataTTLKeeperTimer, rather than implementation. Feel free to raise another PR to add this if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753743522
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
Check the previous exclusive, the tricky thing here is, this repo changed the artifact ID. This is manual override dependency, and keep consistent with our existing dependency.
But the version should be managed in the bom.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753763797
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBStorageProvider.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.storage.IBatchDAO;
+import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
+import org.apache.skywalking.oap.server.core.storage.StorageDAO;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.StorageModule;
+import org.apache.skywalking.oap.server.core.storage.cache.INetworkAddressAliasDAO;
+import org.apache.skywalking.oap.server.core.storage.management.UITemplateManagementDAO;
+import org.apache.skywalking.oap.server.core.storage.model.ModelCreator;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileTaskLogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileTaskQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IAlarmQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ILogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IMetadataQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IMetricsQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITopNRecordsQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
+import org.apache.skywalking.oap.server.library.module.ModuleConfig;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+import org.apache.skywalking.oap.server.library.module.ModuleProvider;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBBatchDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBStorageDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.cache.IoTDBNetworkAddressAliasDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.management.IoTDBUITemplateManagementDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileTaskLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileTaskQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBAlarmQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBMetadataQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBMetricsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTopNRecordsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTraceQueryDAO;
+import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
+import org.apache.skywalking.oap.server.telemetry.api.HealthCheckMetrics;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsCreator;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsTag;
+
+import java.io.IOException;
+
+@Slf4j
+public class IoTDBStorageProvider extends ModuleProvider {
+ private final IoTDBStorageConfig config;
+ private IoTDBClient client;
+
+ public IoTDBStorageProvider() {
+ config = new IoTDBStorageConfig();
+ }
+
+ @Override
+ public String name() {
+ return "iotdb";
+ }
+
+ @Override
+ public Class<? extends ModuleDefine> module() {
+ return StorageModule.class;
+ }
+
+ @Override
+ public ModuleConfig createConfigBeanIfAbsent() {
+ return config;
+ }
+
+ @Override
+ public void prepare() throws ServiceNotProvidedException {
+ this.registerServiceImplementation(StorageBuilderFactory.class, new StorageBuilderFactory.Default());
+
+ try {
+ client = new IoTDBClient(config);
+ } catch (IOException e) {
+ e.printStackTrace();
Review comment:
ok, I have fix it.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
It still has lz4-1.3.0.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
ok
##########
File path: docs/en/setup/backend/backend-storage.md
##########
@@ -280,6 +280,24 @@ storage:
All connection-related settings, including URL link, username, and password are found in `application.yml`.
Only part of the settings are listed here. Please follow [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool document for full settings.
+## IoTDB
Review comment:
Got it, I will fix it in the next commit
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBInsertRequest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+import org.apache.skywalking.oap.server.library.client.request.InsertRequest;
+import org.apache.skywalking.oap.server.library.client.request.UpdateRequest;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+
+@Getter
+@Setter
+@ToString
+@Slf4j
+public class IoTDBInsertRequest implements InsertRequest, UpdateRequest {
+ private String modelName;
+ private long time;
+ private List<String> indexes;
+ private List<String> indexValues;
+ private List<String> measurements;
+ private List<TSDataType> measurementTypes;
+ private List<Object> measurementValues;
+
+ public <T extends StorageData> IoTDBInsertRequest(String modelName, long time, T storageData,
+ StorageHashMapBuilder<T> storageBuilder) {
+ this.modelName = modelName;
+ this.time = time;
+ indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexValues = new ArrayList<>(indexes.size());
+ Map<String, Object> storageMap = storageBuilder.entity2Storage(storageData);
+
+ indexes.forEach(index -> {
+ if (index.equals(IoTDBClient.ID_IDX)) {
+ indexValues.add(storageData.id());
+ } else if (storageMap.containsKey(index)) {
+ indexValues.add(String.valueOf(storageMap.get(index)));
+ storageMap.remove(index);
+ }
+ });
+
+ // time_bucket has changed to time before calling this method, so remove it from measurements
+ storageMap.remove(IoTDBClient.TIME_BUCKET);
+ // processing illegal value
+ Iterator<Map.Entry<String, Object>> entryIterator = storageMap.entrySet().iterator();
+ while (entryIterator.hasNext()) {
+ Map.Entry<String, Object> entry = entryIterator.next();
+ if (entry.getValue() == null) {
+ entryIterator.remove();
+ }
+ if (entry.getValue() instanceof StorageDataComplexObject) {
+ storageMap.put(entry.getKey(), ((StorageDataComplexObject) entry.getValue()).toStorageData());
+ }
+ }
+
+ measurements = new ArrayList<>(storageMap.keySet());
+ Map<String, TSDataType> columnAndTypeMap = IoTDBTableMetaInfo.get(modelName).getColumnAndTypeMap();
+ measurementTypes = new ArrayList<>(measurements.size());
+ for (String measurement : measurements) {
+ measurementTypes.add(columnAndTypeMap.get(measurement));
+ }
+ measurementValues = new ArrayList<>(storageMap.values());
+
+ // processing illegal measurement
Review comment:
Ok, I will rewrite it to make it clear.
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBTableInstaller.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelInstaller;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+
+public class IoTDBTableInstaller extends ModelInstaller {
Review comment:
I will deal with it these days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715358912
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
Review comment:
We should be using IoTDB client SDK but `tomcat-embed-core` seems to be a server side dependency, can you confirm we can exclude it? I don't want to include unnecessary dependencies which might bring CVE in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753737657
##########
File path: dist-material/release-docs/LICENSE
##########
@@ -394,6 +399,8 @@ The following components are provided under the EPL License. See project link fo
The text of each license is also included at licenses/LICENSE-[project].txt.
logback 1.1.11: https://github.com/qos-ch/logback: EPL 1.0
+ logback-classic 1.2.3: https://github.com/qos-ch/logback: EPL 1.0
Review comment:
Can you please also update the version in the previous line (401) 👆 , it should be 1.2.3 as well
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBStorageProvider.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.storage.IBatchDAO;
+import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
+import org.apache.skywalking.oap.server.core.storage.StorageDAO;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.StorageModule;
+import org.apache.skywalking.oap.server.core.storage.cache.INetworkAddressAliasDAO;
+import org.apache.skywalking.oap.server.core.storage.management.UITemplateManagementDAO;
+import org.apache.skywalking.oap.server.core.storage.model.ModelCreator;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileTaskLogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileTaskQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IAlarmQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ILogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IMetadataQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IMetricsQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITopNRecordsQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
+import org.apache.skywalking.oap.server.library.module.ModuleConfig;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+import org.apache.skywalking.oap.server.library.module.ModuleProvider;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBBatchDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBStorageDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.cache.IoTDBNetworkAddressAliasDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.management.IoTDBUITemplateManagementDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileTaskLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileTaskQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.profile.IoTDBProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBAlarmQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBMetadataQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBMetricsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTopNRecordsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.query.IoTDBTraceQueryDAO;
+import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
+import org.apache.skywalking.oap.server.telemetry.api.HealthCheckMetrics;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsCreator;
+import org.apache.skywalking.oap.server.telemetry.api.MetricsTag;
+
+import java.io.IOException;
+
+@Slf4j
+public class IoTDBStorageProvider extends ModuleProvider {
+ private final IoTDBStorageConfig config;
+ private IoTDBClient client;
+
+ public IoTDBStorageProvider() {
+ config = new IoTDBStorageConfig();
+ }
+
+ @Override
+ public String name() {
+ return "iotdb";
+ }
+
+ @Override
+ public Class<? extends ModuleDefine> module() {
+ return StorageModule.class;
+ }
+
+ @Override
+ public ModuleConfig createConfigBeanIfAbsent() {
+ return config;
+ }
+
+ @Override
+ public void prepare() throws ServiceNotProvidedException {
+ this.registerServiceImplementation(StorageBuilderFactory.class, new StorageBuilderFactory.Default());
+
+ try {
+ client = new IoTDBClient(config);
+ } catch (IOException e) {
+ e.printStackTrace();
Review comment:
Never do this in production code, @codeglzhang will you be able to add this prohibition any soon? As we discussed in the other PR
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String GROUP_IDX = "service_group";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) throws IOException {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug(request.toString());
Review comment:
Can you add some descriptive message to the log so that we can search them when needed, such as
```suggestion
log.debug("Writing data to IoTDB: {}", request);
```
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
I don't see there is `lz4` in `iotdb-session`'s dependency tree, please recheck
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBInsertRequest.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+import org.apache.skywalking.oap.server.library.client.request.InsertRequest;
+import org.apache.skywalking.oap.server.library.client.request.UpdateRequest;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+
+@Getter
+@Setter
+@Slf4j
+public class IoTDBInsertRequest implements InsertRequest, UpdateRequest {
+ private String modelName;
+ private long time;
+ private List<String> indexes;
+ private List<String> indexValues;
+ private List<String> measurements;
+ private List<TSDataType> measurementTypes;
+ private List<Object> measurementValues;
+
+ public <T extends StorageData> IoTDBInsertRequest(String modelName, long time, T storageData,
+ StorageHashMapBuilder<T> storageBuilder) {
+ this.modelName = modelName;
+ this.time = time;
+ indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexValues = new ArrayList<>(indexes.size());
+ Map<String, Object> storageMap = storageBuilder.entity2Storage(storageData);
+
+ indexes.forEach(index -> {
+ if (index.equals(IoTDBClient.ID_IDX)) {
+ indexValues.add(storageData.id());
+ } else if (storageMap.containsKey(index)) {
+ indexValues.add(String.valueOf(storageMap.get(index)));
+ storageMap.remove(index);
+ }
+ });
+
+ // time_bucket has changed to time before calling this method, so remove it from measurements
+ storageMap.remove(IoTDBClient.TIME_BUCKET);
+ // processing illegal value
+ Iterator<Map.Entry<String, Object>> entryIterator = storageMap.entrySet().iterator();
+ while (entryIterator.hasNext()) {
+ Map.Entry<String, Object> entry = entryIterator.next();
+ if (entry.getValue() == null) {
+ entryIterator.remove();
+ }
+ if (entry.getValue() instanceof StorageDataComplexObject) {
+ storageMap.put(entry.getKey(), ((StorageDataComplexObject) entry.getValue()).toStorageData());
+ }
+ }
+
+ measurements = new ArrayList<>(storageMap.keySet());
+ Map<String, TSDataType> columnAndTypeMap = IoTDBTableMetaInfo.get(modelName).getColumnAndTypeMap();
+ measurementTypes = new ArrayList<>(measurements.size());
+ for (String measurement : measurements) {
+ measurementTypes.add(columnAndTypeMap.get(measurement));
+ }
+ measurementValues = new ArrayList<>(storageMap.values());
+
+ // processing illegal measurement
+ for (String key : storageMap.keySet()) {
+ if (key.equals(IoTDBClient.TIMESTAMP) || key.contains(".")) {
+ int idx = measurements.indexOf(key);
+ measurements.set(idx, "\"" + key + "\"");
+ }
+ }
+ }
+
+ @Override
+ public String toString() {
Review comment:
Use `@lombok.ToString` instead
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
Review comment:
```suggestion
```
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBTableMetaInfo.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import com.google.gson.JsonObject;
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Getter;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.analysis.NodeType;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+@Getter
+@Builder
+@AllArgsConstructor
+public class IoTDBTableMetaInfo {
+ private static final Map<String, IoTDBTableMetaInfo> TABLE_META_INFOS = new HashMap<>();
+
+ private final Model model;
+ private final Map<String, TSDataType> columnAndTypeMap;
+ private final List<String> indexes;
+
+ public static void addModel(Model model) {
+ final List<ModelColumn> columns = model.getColumns();
+ final Map<String, String> storageAndIndexMap = new HashMap<>();
+ final Map<String, TSDataType> columnAndTypeMap = new HashMap<>();
+ final List<String> indexes = new ArrayList<>();
+
+ storageAndIndexMap.put(model.getName(), IoTDBClient.ID_IDX);
+ columns.forEach(column -> {
+ String columnName = column.getColumnName().getName();
+ if (columnName.equals(IoTDBClient.ENTITY_ID_IDX) || columnName.equals(IoTDBClient.NODE_TYPE_IDX) ||
+ columnName.equals(IoTDBClient.SERVICE_ID_IDX) || columnName.equals(IoTDBClient.GROUP_IDX) ||
+ columnName.equals(IoTDBClient.TRACE_ID_IDX)) {
+ storageAndIndexMap.put(column.getColumnName().getStorageName(), columnName);
+ } else {
+ columnAndTypeMap.put(columnName, typeToTSDataType(column.getType()));
+ }
+ });
+
+ // TODO optimize map to list
Review comment:
Seems this is done?
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
I mean there is no `lz4`, even `net.jpountz.lz4:lz4`, in `iotdb-session`
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
OK, sorry @LIU-WEI-git , it does indeed include an old version of `lz4`, please add the `exclusion` back and move the new `lz4-java` to the `bom` module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755789827
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBEventQueryDAO.java
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.query.PaginationUtils;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.type.event.EventQueryCondition;
+import org.apache.skywalking.oap.server.core.query.type.event.EventType;
+import org.apache.skywalking.oap.server.core.query.type.event.Events;
+import org.apache.skywalking.oap.server.core.query.type.event.Source;
+import org.apache.skywalking.oap.server.core.source.Event;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBEventQueryDAO implements IEventQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<Event> storageBuilder = new Event.Builder();
+
+ @Override
+ public Events queryEvents(EventQueryCondition condition) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(condition, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = page.getFrom(); i < storageDataList.size(); i++) {
+ if (limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ events.getEvents().add(parseEvent(event));
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
+ (org.apache.skywalking.oap.server.core.query.type.event.Event e1,
+ org.apache.skywalking.oap.server.core.query.type.event.Event e2)
+ -> Long.compare(e2.getStartTime(), e1.getStartTime()));
+ } else {
+ events.getEvents().sort(
+ Comparator.comparingLong(org.apache.skywalking.oap.server.core.query.type.event.Event::getStartTime));
+ }
+ return events;
+ }
+
+ @Override
+ public Events queryEvents(List<EventQueryCondition> conditionList) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(conditionList, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ EventQueryCondition condition = conditionList.get(0);
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = page.getFrom(); i < storageDataList.size(); i++) {
+ if (limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ events.getEvents().add(parseEvent(event));
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
+ (org.apache.skywalking.oap.server.core.query.type.event.Event e1,
+ org.apache.skywalking.oap.server.core.query.type.event.Event e2)
+ -> Long.compare(e2.getStartTime(), e1.getStartTime()));
+ } else {
+ events.getEvents().sort(
+ Comparator.comparingLong(org.apache.skywalking.oap.server.core.query.type.event.Event::getStartTime));
+ }
+ return events;
+ }
+
+ private StringBuilder whereSQL(final EventQueryCondition condition, StringBuilder query) {
+ query.append(" where 1=1");
+ if (!Strings.isNullOrEmpty(condition.getUuid())) {
+ query.append(" and ").append(Event.UUID).append(" = \"").append(condition.getUuid()).append("\"");
+ }
+ final Source source = condition.getSource();
+ if (source != null) {
+ if (!Strings.isNullOrEmpty(source.getService())) {
+ query.append(" and ").append(Event.SERVICE).append(" = \"").append(source.getService()).append("\"");
+ }
+ if (!Strings.isNullOrEmpty(source.getServiceInstance())) {
+ query.append(" and ").append(Event.SERVICE_INSTANCE).append(" = \"").append(source.getServiceInstance()).append("\"");
+ }
+ if (!Strings.isNullOrEmpty(source.getEndpoint())) {
+ query.append(" and ").append(Event.ENDPOINT).append(" = \"").append(source.getEndpoint()).append("\"");
+ }
+ }
+ if (!Strings.isNullOrEmpty(condition.getName())) {
+ query.append(" and ").append(Event.NAME).append(" = \"").append(condition.getName()).append("\"");
+ }
+ if (condition.getType() != null) {
+ query.append(" and ").append(Event.TYPE).append(" = \"").append(condition.getType().name()).append("\"");
+ }
+ final Duration time = condition.getTime();
+ if (time != null) {
+ if (time.getStartTimestamp() > 0) {
+ query.append(" and ").append(Event.START_TIME).append(" > ").append(time.getStartTimestamp());
+ }
+ if (time.getEndTimestamp() > 0) {
+ query.append(" and ").append(Event.END_TIME).append(" < ").append(time.getEndTimestamp());
+ }
+ }
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString();
+ queryString = queryString.replace("1=1 and ", "");
+ return new StringBuilder(queryString);
+ }
Review comment:
Thanks for your suggestion. I will deal with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-960426744
```
--- /dev/fd/63 2021-11-04 03:22:37.362685756 +0000
+++ /dev/fd/62 2021-11-04 03:22:37.362685756 +0000
@@ -66 +65,0 @@
-istack-commons-runtime-4.0.0.jar
@@ -74,2 +72,0 @@
-jakarta.activation-2.0.0.jar
-jakarta.xml.bind-api-3.0.0.jar
@@ -78 +74,0 @@
-javax.activation-api-1.2.0.jar
@@ -81,3 +76,0 @@
-jaxb-api-2.4.0-b180830.0359.jar
-jaxb-core-3.0.0.jar
-jaxb-runtime-3.0.0.jar
@@ -158 +150,0 @@
-twx2-3.0.0.jar
```
I think there is some missing about dependency libs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-924699431
Notice this https://github.com/apache/skywalking/tree/master/tools/dependencies. CI will fail due to new dependencies added or changed. This is the way we make sure licenses are checked and managed. You need to update `known-oap-backend-dependencies.txt` with License/Notice by following ASF policy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714953570
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBBrowserLogQueryDAO.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.util.StringUtil;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.browser.source.BrowserErrorCategory;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLog;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLogs;
+import org.apache.skywalking.oap.server.core.query.type.ErrorCategory;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBBrowserLogQueryDAO implements IBrowserLogQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<BrowserErrorLogRecord> storageBuilder = new BrowserErrorLogRecord.Builder();
+
+ @Override
+ public BrowserErrorLogs queryBrowserErrorLogs(String serviceId, String serviceVersionId, String pagePathId,
+ BrowserErrorCategory category, long startSecondTB,
+ long endSecondTB, int limit, int from) throws IOException {
+ StringBuilder query = new StringBuilder();
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query.append("select * from ");
+ query = client.addModelPath(query, BrowserErrorLogRecord.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ if (StringUtil.isNotEmpty(serviceId)) {
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ }
+ query = client.addQueryIndexValue(BrowserErrorLogRecord.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where 1=1");
+ if (startSecondTB != 0 && endSecondTB != 0) {
+ query.append(" and ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startSecondTB));
+ query.append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endSecondTB));
+ }
+ if (StringUtil.isNotEmpty(serviceVersionId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.SERVICE_VERSION_ID).append(" = \"").append(serviceVersionId).append("\"");
+ }
+ if (StringUtil.isNotEmpty(pagePathId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.PAGE_PATH_ID).append(" = \"").append(pagePathId).append("\"");
+ }
+ if (Objects.nonNull(category)) {
+ query.append(" and ").append(BrowserErrorLogRecord.ERROR_CATEGORY).append(" = ").append(category.getValue());
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString();
+ queryString = queryString.replace("1=1 and ", "");
+
+ List<? super StorageData> storageDataList = client.filterQuery(BrowserErrorLogRecord.INDEX_NAME, queryString, storageBuilder);
+ List<BrowserErrorLogRecord> browserErrorLogRecordList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> browserErrorLogRecordList.add((BrowserErrorLogRecord) storageData));
+ // resort by self, because of the select query result order by time.
+ browserErrorLogRecordList.sort((BrowserErrorLogRecord b1, BrowserErrorLogRecord b2) -> Long.compare(b2.getTimestamp(), b1.getTimestamp()));
+ BrowserErrorLogs logs = new BrowserErrorLogs();
+ int limitCount = 0;
+ for (int i = 0; i < browserErrorLogRecordList.size(); i++) {
+ if (i >= from && limitCount < limit) {
+ limitCount++;
+ BrowserErrorLogRecord record = browserErrorLogRecordList.get(i);
+ if (CollectionUtils.isNotEmpty(record.getDataBinary())) {
+ BrowserErrorLog log = iotdbParserDataBinary(record.getDataBinary());
+ logs.getLogs().add(log);
+ }
+ }
+ }
+ logs.setTotal(storageDataList.size());
+ return logs;
+ }
+
+ private BrowserErrorLog iotdbParserDataBinary(byte[] dataBinaryBase64) {
+ try {
+ BrowserErrorLog log = new BrowserErrorLog();
+ org.apache.skywalking.apm.network.language.agent.v3.BrowserErrorLog browserErrorLog = org.apache.skywalking.apm.network.language.agent.v3.BrowserErrorLog
+ .parseFrom(dataBinaryBase64);
+ log.setService(browserErrorLog.getService());
+ log.setServiceVersion(browserErrorLog.getServiceVersion());
+ log.setTime(browserErrorLog.getTime());
+ log.setPagePath(browserErrorLog.getPagePath());
+ log.setCategory(ErrorCategory.valueOf(browserErrorLog.getCategory().name().toUpperCase()));
+ log.setGrade(browserErrorLog.getGrade());
+ log.setMessage(browserErrorLog.getMessage());
+ log.setLine(browserErrorLog.getLine());
+ log.setCol(browserErrorLog.getCol());
+ log.setStack(browserErrorLog.getStack());
+ log.setErrorUrl(browserErrorLog.getErrorUrl());
+ log.setFirstReportedError(browserErrorLog.getFirstReportedError());
+
+ return log;
+ } catch (InvalidProtocolBufferException e) {
Review comment:
No. This is method is refer to `IBrowserLogQueryDAO.parserDataBinary` without Base64 decode, because it has been decoded in `BrowserErrorLogRecord.Builder.storage2Entity`. The InvalidProtocolBufferException is thrown by `org.apache.skywalking.apm.network.language.agent.v3.BrowserErrorLog.parseFrom`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r719052784
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,62 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.tomcat.embed</groupId>
+ <artifactId>tomcat-embed-core</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
Review comment:
*Critical OSS Vulnerability:*
### pkg:maven/org.apache.tomcat.embed/tomcat-embed-core
57 Critical, 117 Severe, 13 Moderate, 0 Unknown vulnerabilities have been found across 1 dependencies
<details>
<summary><b>Components</b></summary><br/>
<ul>
<details>
<summary><b>pkg:maven/org.apache.tomcat.embed/tomcat-embed-core</b></summary>
<ul>
<details>
<summary><b>CRITICAL Vulnerabilities (57)</b></summary><br/>
<ul>
<details>
<summary>CVE-2016-8735</summary>
> #### [CVE-2016-8735] Improper Access Control
> Remote code execution is possible with Apache Tomcat before 6.0.48, 7.x before 7.0.73, 8.x before 8.0.39, 8.5.x before 8.5.7, and 9.x before 9.0.0.M12 if JmxRemoteLifecycleListener is used and an attacker can reach JMX ports. The issue exists because this listener wasn't updated for consistency with the CVE-2016-3427 Oracle patch that affected credential types.
>
> **CVSS Score:** 9.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2017-5651</summary>
> #### [CVE-2017-5651] Data Handling
> In Apache Tomcat 9.0.0.M1 to 9.0.0.M18 and 8.5.0 to 8.5.12, the refactoring of the HTTP connectors introduced a regression in the send file processing. If the send file processing completed quickly, it was possible for the Processor to be added to the processor cache twice. This could result in the same Processor being used for multiple requests which in turn could lead to unexpected errors and/or response mix-up.
>
> **CVSS Score:** 9.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2018-8014</summary>
> #### [CVE-2018-8014] The defaults settings for the CORS filter provided in Apache Tomcat 9.0.0.M1 to ...
> The defaults settings for the CORS filter provided in Apache Tomcat 9.0.0.M1 to 9.0.8, 8.5.0 to 8.5.31, 8.0.0.RC1 to 8.0.52, 7.0.41 to 7.0.88 are insecure and enable 'supportsCredentials' for all origins. It is expected that users of the CORS filter will have configured it appropriately for their environment rather than using it in the default configuration. Therefore, it is expected that most users will not be impacted by this issue.
>
> **CVSS Score:** 9.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2020-1938</summary>
> #### [CVE-2020-1938] When using the Apache JServ Protocol (AJP), care must be taken when trusting inc...
> When using the Apache JServ Protocol (AJP), care must be taken when trusting incoming connections to Apache Tomcat. Tomcat treats AJP connections as having higher trust than, for example, a similar HTTP connection. If such connections are available to an attacker, they can be exploited in ways that may be surprising. In Apache Tomcat 9.0.0.M1 to 9.0.0.30, 8.5.0 to 8.5.50 and 7.0.0 to 7.0.99, Tomcat shipped with an AJP Connector enabled by default that listened on all configured IP addresses. It was expected (and recommended in the security guide) that this Connector would be disabled if not required. This vulnerability report identified a mechanism that allowed: - returning arbitrary files from anywhere in the web application - processing any file in the web application as a JSP Further, if the web application allowed file upload and stored those files within the web application (or the attacker was able to control the content of the web application by some other means) then thi
s, along with the ability to process a file as a JSP, made remote code execution possible. It is important to note that mitigation is only required if an AJP port is accessible to untrusted users. Users wishing to take a defence-in-depth approach and block the vector that permits returning arbitrary files and execution as JSP may upgrade to Apache Tomcat 9.0.31, 8.5.51 or 7.0.100 or later. A number of changes were made to the default AJP Connector configuration in 9.0.31 to harden the default configuration. It is likely that users upgrading to 9.0.31, 8.5.51 or 7.0.100 or later will need to make small changes to their configurations.
>
> **CVSS Score:** 9.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2017-5648</summary>
> #### [CVE-2017-5648] Improper Access Control
> While investigating bug 60718, it was noticed that some calls to application listeners in Apache Tomcat 9.0.0.M1 to 9.0.0.M17, 8.5.0 to 8.5.11, 8.0.0.RC1 to 8.0.41, and 7.0.0 to 7.0.75 did not use the appropriate facade object. When running an untrusted application under a SecurityManager, it was therefore possible for that untrusted application to retain a reference to the request or response object and thereby access and/or modify information associated with another web application.
>
> **CVSS Score:** 9.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N
</details>
<details>
<summary>CVE-2016-0714</summary>
> #### [CVE-2016-0714] Permissions, Privileges, and Access Controls
> The session-persistence implementation in Apache Tomcat 6.x before 6.0.45, 7.x before 7.0.68, 8.x before 8.0.31, and 9.x before 9.0.0.M2 mishandles session attributes, which allows remote authenticated users to bypass intended SecurityManager restrictions and execute arbitrary code in a privileged context via a web application that places a crafted object in a session.
>
> **CVSS Score:** 8.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2015-5351</summary>
> #### [CVE-2015-5351] Cross-Site Request Forgery (CSRF)
> The (1) Manager and (2) Host Manager applications in Apache Tomcat 7.x before 7.0.68, 8.x before 8.0.31, and 9.x before 9.0.0.M2 establish sessions and send CSRF tokens for arbitrary new requests, which allows remote attackers to bypass a CSRF protection mechanism by using a token.
>
> **CVSS Score:** 8.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2021-30640</summary>
> #### [CVE-2021-30640] A vulnerability in the JNDI Realm of Apache Tomcat allows an attacker to authent...
> A vulnerability in the JNDI Realm of Apache Tomcat allows an attacker to authenticate using variations of a valid user name and/or to bypass some of the protection provided by the LockOut Realm. This issue affects Apache Tomcat 10.0.0-M1 to 10.0.5; 9.0.0.M1 to 9.0.45; 8.5.0 to 8.5.65.
>
> **CVSS Score:** 8.2
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:H/A:N
</details>
<details>
<summary>CVE-2019-0232</summary>
> #### [CVE-2019-0232] Improper Input Validation
> When running on Windows with enableCmdLineArguments enabled, the CGI Servlet in Apache Tomcat 9.0.0.M1 to 9.0.17, 8.5.0 to 8.5.39 and 7.0.0 to 7.0.93 is vulnerable to Remote Code Execution due to a bug in the way the JRE passes command line arguments to Windows. The CGI Servlet is disabled by default. The CGI option enableCmdLineArguments is disable by default in Tomcat 9.0.x (and will be disabled by default in all versions in response to this vulnerability). For a detailed explanation of the JRE behaviour, see Markus Wulftange's blog (https://codewhitesec.blogspot.com/2016/02/java-and-command-line-injections-in-windows.html) and this archived MSDN blog (https://web.archive.org/web/20161228144344/https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/everyone-quotes-command-line-arguments-the-wrong-way/).
>
> **CVSS Score:** 8.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2017-12615</summary>
> #### [CVE-2017-12615] When running Apache Tomcat 7.0.0 to 7.0.79 on Windows with HTTP PUTs enabled (e....
> When running Apache Tomcat 7.0.0 to 7.0.79 on Windows with HTTP PUTs enabled (e.g. via setting the readonly initialisation parameter of the Default to false) it was possible to upload a JSP file to the server via a specially crafted request. This JSP could then be requested and any code it contained would be executed by the server.
>
> **CVSS Score:** 8.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2017-12617</summary>
> #### [CVE-2017-12617] When running Apache Tomcat versions 9.0.0.M1 to 9.0.0, 8.5.0 to 8.5.22, 8.0.0.RC...
> When running Apache Tomcat versions 9.0.0.M1 to 9.0.0, 8.5.0 to 8.5.22, 8.0.0.RC1 to 8.0.46 and 7.0.0 to 7.0.81 with HTTP PUTs enabled (e.g. via setting the readonly initialisation parameter of the Default servlet to false) it was possible to upload a JSP file to the server via a specially crafted request. This JSP could then be requested and any code it contained would be executed by the server.
>
> **CVSS Score:** 8.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2015-5346</summary>
> #### [CVE-2015-5346] Session fixation vulnerability in Apache Tomcat 7.x before 7.0.66, 8.x before 8....
> Session fixation vulnerability in Apache Tomcat 7.x before 7.0.66, 8.x before 8.0.30, and 9.x before 9.0.0.M2, when different session settings are used for deployments of multiple versions of the same web application, might allow remote attackers to hijack web sessions by leveraging use of a requestedSessionSSL field for an unintended request, related to CoyoteAdapter.java and Request.java.
>
> **CVSS Score:** 8.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2016-5388</summary>
> #### [CVE-2016-5388] Improper Access Control
> Apache Tomcat through 8.5.4, when the CGI Servlet is enabled, follows RFC 3875 section 4.1.18 and therefore does not protect applications from the presence of untrusted client data in the HTTP_PROXY environment variable, which might allow remote attackers to redirect an application's outbound HTTP traffic to an arbitrary proxy server via a crafted Proxy header in an HTTP request, aka an "httpoxy" issue. NOTE: the vendor states "A mitigation is planned for future releases of Tomcat, tracked as CVE-2016-5388"; in other words, this is not a CVE ID for a vulnerability.
>
> **CVSS Score:** 8.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2002-2272</summary>
> #### [CVE-2002-2272] Improper Restriction of Operations within the Bounds of a Memory Buffer
> Tomcat 4.0 through 4.1.12, using mod_jk 1.2.1 module on Apache 1.3 through 1.3.27, allows remote attackers to cause a denial of service (desynchronized communications) via an HTTP GET request with a Transfer-Encoding chunked field with invalid values.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:C
</details>
<details>
<summary>CVE-2005-4836</summary>
> #### [CVE-2005-4836] Information Exposure
> The HTTP/1.1 connector in Apache Tomcat 4.1.15 through 4.1.40 does not reject NULL bytes in a URL when allowLinking is configured, which allows remote attackers to read JSP source files and obtain sensitive information.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:C/I:N/A:N
</details>
<details>
<summary>CVE-2006-7197</summary>
> #### [CVE-2006-7197] The AJP connector in Apache Tomcat 5.5.15 uses an incorrect length for chunks, w...
> The AJP connector in Apache Tomcat 5.5.15 uses an incorrect length for chunks, which can cause a buffer over-read in the ajp_process_callback in mod_jk, which allows remote attackers to read portions of sensitive memory.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:C/I:N/A:N
</details>
<details>
<summary>CVE-2016-9774</summary>
> #### [CVE-2016-9774] Improper Link Resolution Before File Access ("Link Following")
> The postinst script in the tomcat6 package before 6.0.45+dfsg-1~deb7u4 on Debian wheezy, before 6.0.35-1ubuntu3.9 on Ubuntu 12.04 LTS and on Ubuntu 14.04 LTS; the tomcat7 package before 7.0.28-4+deb7u8 on Debian wheezy, before 7.0.56-3+deb8u6 on Debian jessie, before 7.0.52-1ubuntu0.8 on Ubuntu 14.04 LTS, and on Ubuntu 12.04 LTS, 16.04 LTS, and 16.10; and the tomcat8 package before 8.0.14-1+deb8u5 on Debian jessie, before 8.0.32-1ubuntu1.3 on Ubuntu 16.04 LTS, before 8.0.37-1ubuntu0.1 on Ubuntu 16.10, and before 8.0.38-2ubuntu1 on Ubuntu 17.04 might allow local users with access to the tomcat account to obtain sensitive information or gain root privileges via a symlink attack on the Catalina localhost directory.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2014-0230</summary>
> #### [CVE-2014-0230] Resource Management Errors
> Apache Tomcat 6.x before 6.0.44, 7.x before 7.0.55, and 8.x before 8.0.9 does not properly handle cases where an HTTP response occurs before finishing the reading of an entire request body, which allows remote attackers to cause a denial of service (thread consumption) via a series of aborted upload attempts.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:C
</details>
<details>
<summary>CVE-2016-1240</summary>
> #### [CVE-2016-1240] Improper Input Validation
> The Tomcat init script in the tomcat7 package before 7.0.56-3+deb8u4 and tomcat8 package before 8.0.14-1+deb8u3 on Debian jessie and the tomcat6 and libtomcat6-java packages before 6.0.35-1ubuntu3.8 on Ubuntu 12.04 LTS, the tomcat7 and libtomcat7-java packages before 7.0.52-1ubuntu0.7 on Ubuntu 14.04 LTS, and tomcat8 and libtomcat8-java packages before 8.0.32-1ubuntu1.2 on Ubuntu 16.04 LTS allows local users with access to the tomcat account to gain root privileges via a symlink attack on the Catalina log file, as demonstrated by /var/log/tomcat7/catalina.out.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2016-9775</summary>
> #### [CVE-2016-9775] Permissions, Privileges, and Access Controls
> The postrm script in the tomcat6 package before 6.0.45+dfsg-1~deb7u3 on Debian wheezy, before 6.0.45+dfsg-1~deb8u1 on Debian jessie, before 6.0.35-1ubuntu3.9 on Ubuntu 12.04 LTS and on Ubuntu 14.04 LTS; the tomcat7 package before 7.0.28-4+deb7u7 on Debian wheezy, before 7.0.56-3+deb8u6 on Debian jessie, before 7.0.52-1ubuntu0.8 on Ubuntu 14.04 LTS, and on Ubuntu 12.04 LTS, 16.04 LTS, and 16.10; and the tomcat8 package before 8.0.14-1+deb8u5 on Debian jessie, before 8.0.32-1ubuntu1.3 on Ubuntu 16.04 LTS, before 8.0.37-1ubuntu0.1 on Ubuntu 16.10, and before 8.0.38-2ubuntu1 on Ubuntu 17.04 might allow local users with access to the tomcat account to gain root privileges via a setgid program in the Catalina directory, as demonstrated by /etc/tomcat8/Catalina/attack.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2002-0493</summary>
> #### [CVE-2002-0493] null
> Apache Tomcat may be started without proper security settings if errors are encountered while reading the web.xml file, which could allow attackers to bypass intended restrictions.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2018-8034</summary>
> #### [CVE-2018-8034] The host name verification when using TLS with the WebSocket client was missing....
> The host name verification when using TLS with the WebSocket client was missing. It is now enabled by default. Versions Affected: Apache Tomcat 9.0.0.M1 to 9.0.9, 8.5.0 to 8.5.31, 8.0.0.RC1 to 8.0.52, and 7.0.35 to 7.0.88.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2017-5664</summary>
> #### [CVE-2017-5664] The error page mechanism of the Java Servlet Specification requires that, when a...
> The error page mechanism of the Java Servlet Specification requires that, when an error occurs and an error page is configured for the error that occurred, the original request and response are forwarded to the error page. This means that the request is presented to the error page with the original HTTP method. If the error page is a static file, expected behaviour is to serve content of the file as if processing a GET request, regardless of the actual HTTP method. The Default Servlet in Apache Tomcat 9.0.0.M1 to 9.0.0.M20, 8.5.0 to 8.5.14, 8.0.0.RC1 to 8.0.43 and 7.0.0 to 7.0.77 did not do this. Depending on the original request this could lead to unexpected and undesirable results for static error pages including, if the DefaultServlet is configured to permit writes, the replacement or removal of the custom error page. Notes for other user provided error pages: (1) Unless explicitly coded otherwise, JSPs ignore the HTTP method. JSPs used as error pages must must ensure that t
hey handle any error dispatch as a GET request, regardless of the actual method. (2) By default, the response generated by a Servlet does depend on the HTTP method. Custom Servlets used as error pages must ensure that they handle any error dispatch as a GET request, regardless of the actual method.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N
</details>
<details>
<summary>CVE-2016-6797</summary>
> #### [CVE-2016-6797] Improper Access Control
> The ResourceLinkFactory implementation in Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0.0 to 7.0.70 and 6.0.0 to 6.0.45 did not limit web application access to global JNDI resources to those resources explicitly linked to the web application. Therefore, it was possible for a web application to access any global JNDI resource whether an explicit ResourceLink had been configured or not.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2002-1394</summary>
> #### [CVE-2002-1394] Apache Tomcat 4.0.5 and earlier, when using both the invoker servlet and the def...
> Apache Tomcat 4.0.5 and earlier, when using both the invoker servlet and the default servlet, allows remote attackers to read source code for server files or bypass certain protections, a variant of CAN-2002-1148.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2016-3092</summary>
> #### [CVE-2016-3092] Improper Input Validation
> The MultipartStream class in Apache Commons Fileupload before 1.3.2, as used in Apache Tomcat 7.x before 7.0.70, 8.x before 8.0.36, 8.5.x before 8.5.3, and 9.x before 9.0.0.M7 and other products, allows remote attackers to cause a denial of service (CPU consumption) via a long boundary string.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2016-6817</summary>
> #### [CVE-2016-6817] Improper Restriction of Operations within the Bounds of a Memory Buffer
> The HTTP/2 header parser in Apache Tomcat 9.0.0.M1 to 9.0.0.M11 and 8.5.0 to 8.5.6 entered an infinite loop if a header was received that was larger than the available buffer. This made a denial of service attack possible.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2014-0050</summary>
> #### [CVE-2014-0050] Permissions, Privileges, and Access Controls
> MultipartStream.java in Apache Commons FileUpload before 1.3.1, as used in Apache Tomcat, JBoss Web, and other products, allows remote attackers to cause a denial of service (infinite loop and CPU consumption) via a crafted Content-Type header that bypasses a loop's intended exit conditions.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2017-12616</summary>
> #### [CVE-2017-12616] When using a VirtualDirContext with Apache Tomcat 7.0.0 to 7.0.80 it was possibl...
> When using a VirtualDirContext with Apache Tomcat 7.0.0 to 7.0.80 it was possible to bypass security constraints and/or view the source code of JSPs for resources served by the VirtualDirContext using a specially crafted request.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2020-13935</summary>
> #### [CVE-2020-13935] The payload length in a WebSocket frame was not correctly validated in Apache To...
> The payload length in a WebSocket frame was not correctly validated in Apache Tomcat 10.0.0-M1 to 10.0.0-M6, 9.0.0.M1 to 9.0.36, 8.5.0 to 8.5.56 and 7.0.27 to 7.0.104. Invalid payload lengths could trigger an infinite loop. Multiple requests with invalid payload lengths could lead to a denial of service.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2019-17563</summary>
> #### [CVE-2019-17563] When using FORM authentication with Apache Tomcat 9.0.0.M1 to 9.0.29, 8.5.0 to 8...
> When using FORM authentication with Apache Tomcat 9.0.0.M1 to 9.0.29, 8.5.0 to 8.5.49 and 7.0.0 to 7.0.98 there was a narrow window where an attacker could perform a session fixation attack. The window was considered too narrow for an exploit to be practical but, erring on the side of caution, this issue has been treated as a security vulnerability.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:R/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2016-6796</summary>
> #### [CVE-2016-6796] A malicious web application running on Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0...
> A malicious web application running on Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0.0 to 7.0.70 and 6.0.0 to 6.0.45 was able to bypass a configured SecurityManager via manipulation of the configuration parameters for the JSP Servlet.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N
</details>
<details>
<summary>CVE-2017-5650</summary>
> #### [CVE-2017-5650] Resource Management Errors
> In Apache Tomcat 9.0.0.M1 to 9.0.0.M18 and 8.5.0 to 8.5.12, the handling of an HTTP/2 GOAWAY frame for a connection did not close streams associated with that connection that were currently waiting for a WINDOW_UPDATE before allowing the application to write more data. These waiting streams each consumed a thread. A malicious client could therefore construct a series of HTTP/2 requests that would consume all available processing threads.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2020-13934</summary>
> #### [CVE-2020-13934] An h2c direct connection to Apache Tomcat 10.0.0-M1 to 10.0.0-M6, 9.0.0.M5 to 9....
> An h2c direct connection to Apache Tomcat 10.0.0-M1 to 10.0.0-M6, 9.0.0.M5 to 9.0.36 and 8.5.1 to 8.5.56 did not release the HTTP/1.1 processor after the upgrade to HTTP/2. If a sufficient number of such requests were made, an OutOfMemoryException could occur leading to a denial of service.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2017-7675</summary>
> #### [CVE-2017-7675] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> The HTTP/2 implementation in Apache Tomcat 9.0.0.M1 to 9.0.0.M21 and 8.5.0 to 8.5.15 bypassed a number of security checks that prevented directory traversal attacks. It was therefore possible to bypass security constraints using a specially crafted URL.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2021-41079</summary>
> #### [CVE-2021-41079] Apache Tomcat 8.5.0 to 8.5.63, 9.0.0-M1 to 9.0.43 and 10.0.0-M1 to 10.0.2 did no...
> Apache Tomcat 8.5.0 to 8.5.63, 9.0.0-M1 to 9.0.43 and 10.0.0-M1 to 10.0.2 did not properly validate incoming TLS packets. When Tomcat was configured to use NIO+OpenSSL or NIO2+OpenSSL for TLS, a specially crafted packet could be used to trigger an infinite loop resulting in a denial of service.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2020-17527</summary>
> #### [CVE-2020-17527] While investigating bug 64830 it was discovered that Apache Tomcat 10.0.0-M1 to ...
> While investigating bug 64830 it was discovered that Apache Tomcat 10.0.0-M1 to 10.0.0-M9, 9.0.0-M1 to 9.0.39 and 8.5.0 to 8.5.59 could re-use an HTTP request header value from the previous stream received on an HTTP/2 connection for the request associated with the subsequent stream. While this would most likely lead to an error and the closure of the HTTP/2 connection, it is possible that information could leak between requests.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2018-1336</summary>
> #### [CVE-2018-1336] Uncontrolled Resource Consumption ("Resource Exhaustion")
> An improper handing of overflow in the UTF-8 decoder with supplementary characters can lead to an infinite loop in the decoder causing a Denial of Service. Versions Affected: Apache Tomcat 9.0.0.M9 to 9.0.7, 8.5.0 to 8.5.30, 8.0.0.RC1 to 8.0.51, and 7.0.28 to 7.0.86.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2013-2185</summary>
> #### [CVE-2013-2185] Improper Input Validation
> ** DISPUTED ** The readObject method in the DiskFileItem class in Apache Tomcat and JBoss Web, as used in Red Hat JBoss Enterprise Application Platform 6.1.0 and Red Hat JBoss Portal 6.0.0, allows remote attackers to write to arbitrary files via a NULL byte in a file name in a serialized instance, a similar issue to CVE-2013-2186. NOTE: this issue is reportedly disputed by the Apache Tomcat team, although Red Hat considers it a vulnerability. The dispute appears to regard whether it is the responsibility of applications to avoid providing untrusted data to be deserialized, or whether this class should inherently protect against this issue.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2020-11996</summary>
> #### [CVE-2020-11996] A specially crafted sequence of HTTP/2 requests sent to Apache Tomcat 10.0.0-M1 ...
> A specially crafted sequence of HTTP/2 requests sent to Apache Tomcat 10.0.0-M1 to 10.0.0-M5, 9.0.0.M1 to 9.0.35 and 8.5.0 to 8.5.55 could trigger high CPU usage for several seconds. If a sufficient number of such requests were made on concurrent HTTP/2 connections, the server could become unresponsive.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2021-25122</summary>
> #### [CVE-2021-25122] When responding to new h2c connection requests, Apache Tomcat versions 10.0.0-M1...
> When responding to new h2c connection requests, Apache Tomcat versions 10.0.0-M1 to 10.0.0, 9.0.0.M1 to 9.0.41 and 8.5.0 to 8.5.61 could duplicate request headers and a limited amount of request body from one request to another meaning user A and user B could both see the results of user A's request.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2001-1563</summary>
> #### [CVE-2001-1563] Unknown vulnerability in Tomcat 3.2.1 running on HP Secure OS for Linux 1.0 allo...
> Unknown vulnerability in Tomcat 3.2.1 running on HP Secure OS for Linux 1.0 allows attackers to access servlet resources. NOTE: due to the vagueness of the vendor advisory, it is not clear whether this issue is already covered by other CVE identifiers.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2009-3548</summary>
> #### [CVE-2009-3548] Credentials Management
> The Windows installer for Apache Tomcat 6.0.0 through 6.0.20, 5.5.0 through 5.5.28, and possibly earlier versions uses a blank default password for the administrative user, which allows remote attackers to gain privileges.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2021-24122</summary>
> #### [CVE-2021-24122] When serving resources from a network location using the NTFS file system, Apach...
> When serving resources from a network location using the NTFS file system, Apache Tomcat versions 10.0.0-M1 to 10.0.0-M9, 9.0.0.M1 to 9.0.39, 8.5.0 to 8.5.59 and 7.0.0 to 7.0.106 were susceptible to JSP source code disclosure in some configurations. The root cause was the unexpected behaviour of the JRE API File.getCanonicalPath() which in turn was caused by the inconsistent behaviour of the Windows API (FindFirstFileW) in some circumstances.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2002-0682</summary>
> #### [CVE-2002-0682] Cross-site scripting vulnerability in Apache Tomcat 4.0.3 allows remote attacker...
> Cross-site scripting vulnerability in Apache Tomcat 4.0.3 allows remote attackers to execute script as other web users via script in a URL with the /servlet/ mapping, which does not filter the script when an exception is thrown by the servlet.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2021-30639</summary>
> #### [CVE-2021-30639] A vulnerability in Apache Tomcat allows an attacker to remotely trigger a denial...
> A vulnerability in Apache Tomcat allows an attacker to remotely trigger a denial of service. An error introduced as part of a change to improve error handling during non-blocking I/O meant that the error flag associated with the Request object was not reset between requests. This meant that once a non-blocking I/O error occurred, all future requests handled by that request object would fail. Users were able to trigger non-blocking I/O errors, e.g. by dropping a connection, thereby creating the possibility of triggering a DoS. Applications that do not use non-blocking I/O are not exposed to this vulnerability. This issue affects Apache Tomcat 10.0.3 to 10.0.4; 9.0.44; 8.5.64.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2016-5018</summary>
> #### [CVE-2016-5018] In Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0....
> In Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0.0 to 7.0.70 and 6.0.0 to 6.0.45 a malicious web application was able to bypass a configured SecurityManager via a Tomcat utility method that was accessible to web applications.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N
</details>
<details>
<summary>CVE-2011-3190</summary>
> #### [CVE-2011-3190] Permissions, Privileges, and Access Controls
> Certain AJP protocol connector implementations in Apache Tomcat 7.0.0 through 7.0.20, 6.0.0 through 6.0.33, 5.5.0 through 5.5.33, and possibly other versions allow remote attackers to spoof AJP requests, bypass authentication, and obtain sensitive information by causing the connector to interpret a request body as a new request.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2019-0199</summary>
> #### [CVE-2019-0199] Uncontrolled Resource Consumption ("Resource Exhaustion")
> The HTTP/2 implementation in Apache Tomcat 9.0.0.M1 to 9.0.14 and 8.5.0 to 8.5.37 accepted streams with excessive numbers of SETTINGS frames and also permitted clients to keep streams open without reading/writing request/response data. By keeping streams open for requests that utilised the Servlet API's blocking I/O, clients were able to cause server-side threads to block eventually leading to thread exhaustion and a DoS.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2017-5647</summary>
> #### [CVE-2017-5647] Information Exposure
> A bug in the handling of the pipelined requests in Apache Tomcat 9.0.0.M1 to 9.0.0.M18, 8.5.0 to 8.5.12, 8.0.0.RC1 to 8.0.42, 7.0.0 to 7.0.76, and 6.0.0 to 6.0.52, when send file was used, results in the pipelined request being lost when send file processing of the previous request completed. This could result in responses appearing to be sent for the wrong request. For example, a user agent that sent requests A, B and C could see the correct response for request A, the response for request C for request B and no response for request C.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2016-8745</summary>
> #### [CVE-2016-8745] A bug in the error handling of the send file code for the NIO HTTP connector in ...
> A bug in the error handling of the send file code for the NIO HTTP connector in Apache Tomcat 9.0.0.M1 to 9.0.0.M13, 8.5.0 to 8.5.8, 8.0.0.RC1 to 8.0.39, 7.0.0 to 7.0.73 and 6.0.16 to 6.0.48 resulted in the current Processor object being added to the Processor cache multiple times. This in turn meant that the same Processor could be used for concurrent requests. Sharing a Processor can result in information leakage between requests including, not not limited to, session ID and the response body. The bug was first noticed in 8.5.x onwards where it appears the refactoring of the Connector code for 8.5.x onwards made it more likely that the bug was observed. Initially it was thought that the 8.5.x refactoring introduced the bug but further investigation has shown that the bug is present in all currently supported Tomcat versions.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2019-10072</summary>
> #### [CVE-2019-10072] Uncontrolled Resource Consumption ("Resource Exhaustion")
> The fix for CVE-2019-0199 was incomplete and did not address HTTP/2 connection window exhaustion on write in Apache Tomcat versions 9.0.0.M1 to 9.0.19 and 8.5.0 to 8.5.40 . By not sending WINDOW_UPDATE messages for the connection window (stream 0) clients were able to cause server-side threads to block eventually leading to thread exhaustion and a DoS.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2016-8747</summary>
> #### [CVE-2016-8747] Information Exposure
> An information disclosure issue was discovered in Apache Tomcat 8.5.7 to 8.5.9 and 9.0.0.M11 to 9.0.0.M15 in reverse-proxy configurations. Http11InputBuffer.java allows remote attackers to read data that was intended to be associated with a different request.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2016-6816</summary>
> #### [CVE-2016-6816] Improper Input Validation
> The code in Apache Tomcat 9.0.0.M1 to 9.0.0.M11, 8.5.0 to 8.5.6, 8.0.0.RC1 to 8.0.38, 7.0.0 to 7.0.72, and 6.0.0 to 6.0.47 that parsed the HTTP request line permitted invalid characters. This could be exploited, in conjunction with a proxy that also permitted the invalid characters but with a different interpretation, to inject data into the HTTP response. By manipulating the HTTP response the attacker could poison a web-cache, perform an XSS attack and/or obtain sensitive information from requests other then their own.
>
> **CVSS Score:** 7.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:C/C:L/I:L/A:L
</details>
<details>
<summary>CVE-2020-9484</summary>
> #### [CVE-2020-9484] When using Apache Tomcat versions 10.0.0-M1 to 10.0.0-M4, 9.0.0.M1 to 9.0.34, 8....
> When using Apache Tomcat versions 10.0.0-M1 to 10.0.0-M4, 9.0.0.M1 to 9.0.34, 8.5.0 to 8.5.54 and 7.0.0 to 7.0.103 if a) an attacker is able to control the contents and name of a file on the server; and b) the server is configured to use the PersistenceManager with a FileStore; and c) the PersistenceManager is configured with sessionAttributeValueClassNameFilter="null" (the default unless a SecurityManager is used) or a sufficiently lax filter to allow the attacker provided object to be deserialized; and d) the attacker knows the relative file path from the storage location used by FileStore to the file the attacker has control over; then, using a specifically crafted request, the attacker will be able to trigger remote code execution via deserialization of the file under their control. Note that all of conditions a) to d) must be true for the attack to succeed.
>
> **CVSS Score:** 7
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2021-25329</summary>
> #### [CVE-2021-25329] The fix for CVE-2020-9484 was incomplete. When using Apache Tomcat 10.0.0-M1 to ...
> The fix for CVE-2020-9484 was incomplete. When using Apache Tomcat 10.0.0-M1 to 10.0.0, 9.0.0.M1 to 9.0.41, 8.5.0 to 8.5.61 or 7.0.0. to 7.0.107 with a configuration edge case that was highly unlikely to be used, the Tomcat instance was still vulnerable to CVE-2020-9494. Note that both the previously published prerequisites for CVE-2020-9484 and the previously published mitigations for CVE-2020-9484 also apply to this issue.
>
> **CVSS Score:** 7
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2019-12418</summary>
> #### [CVE-2019-12418] When Apache Tomcat 9.0.0.M1 to 9.0.28, 8.5.0 to 8.5.47, 7.0.0 and 7.0.97 is conf...
> When Apache Tomcat 9.0.0.M1 to 9.0.28, 8.5.0 to 8.5.47, 7.0.0 and 7.0.97 is configured with the JMX Remote Lifecycle Listener, a local attacker without access to the Tomcat process or configuration files is able to manipulate the RMI registry to perform a man-in-the-middle attack to capture user names and passwords used to access the JMX interface. The attacker can then use these credentials to access the JMX interface and gain complete control over the Tomcat instance.
>
> **CVSS Score:** 7
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
</ul>
</details>
<details>
<summary><b>SEVERE Vulnerabilities (117)</b></summary><br/>
<ul>
<details>
<summary>CVE-2013-6357</summary>
> #### [CVE-2013-6357] Cross-Site Request Forgery (CSRF)
> ** DISPUTED ** Cross-site request forgery (CSRF) vulnerability in the Manager application in Apache Tomcat 5.5.25 and earlier allows remote attackers to hijack the authentication of administrators for requests that manipulate application deployment via the POST method, as demonstrated by a /manager/html/undeploy?path= URI. NOTE: the vendor disputes the significance of this report, stating that "the Apache Tomcat Security team has not accepted any reports of CSRF attacks against the Manager application ... as they require a reckless system administrator."
>
> **CVSS Score:** 6.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2002-1567</summary>
> #### [CVE-2002-1567] Cross-site scripting (XSS) vulnerability in Apache Tomcat 4.1 allows remote atta...
> Cross-site scripting (XSS) vulnerability in Apache Tomcat 4.1 allows remote attackers to execute arbitrary web script and steal cookies via a URL with encoded newlines followed by a request to a .jsp file whose name contains the script.
>
> **CVSS Score:** 6.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2013-2067</summary>
> #### [CVE-2013-2067] Improper Authentication
> java/org/apache/catalina/authenticator/FormAuthenticator.java in the form authentication feature in Apache Tomcat 6.0.21 through 6.0.36 and 7.x before 7.0.33 does not properly handle the relationships between authentication requirements and sessions, which allows remote attackers to inject a request into a session by sending this request during completion of the login form, a variant of a session fixation attack.
>
> **CVSS Score:** 6.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2013-4444</summary>
> #### [CVE-2013-4444] Improper Control of Generation of Code ("Code Injection")
> Unrestricted file upload vulnerability in Apache Tomcat 7.x before 7.0.40, in certain situations involving outdated java.io.File code and a custom JMX configuration, allows remote attackers to execute arbitrary code by uploading and accessing a JSP file.
>
> **CVSS Score:** 6.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2003-0044</summary>
> #### [CVE-2003-0044] Multiple cross-site scripting (XSS) vulnerabilities in the (1) examples and (2) ...
> Multiple cross-site scripting (XSS) vulnerabilities in the (1) examples and (2) ROOT web applications for Jakarta Tomcat 3.x through 3.3.1a allow remote attackers to insert arbitrary web script or HTML.
>
> **CVSS Score:** 6.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2018-1305</summary>
> #### [CVE-2018-1305] Improper Access Control
> Security constraints defined by annotations of Servlets in Apache Tomcat 9.0.0.M1 to 9.0.4, 8.5.0 to 8.5.27, 8.0.0.RC1 to 8.0.49 and 7.0.0 to 7.0.84 were only applied once a Servlet had been loaded. Because security constraints defined in this way apply to the URL pattern and any URLs below that point, it was possible - depending on the order Servlets were loaded - for some security constraints not to be applied. This could have exposed resources to users who were not authorised to access them.
>
> **CVSS Score:** 6.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2010-2227</summary>
> #### [CVE-2010-2227] Improper Restriction of Operations within the Bounds of a Memory Buffer
> Apache Tomcat 5.5.0 through 5.5.29, 6.0.0 through 6.0.27, and 7.0.0 beta does not properly handle an invalid Transfer-Encoding header, which allows remote attackers to cause a denial of service (application outage) or obtain sensitive information via a crafted header that interferes with "recycling of a buffer."
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:P
</details>
<details>
<summary>CVE-2007-5342</summary>
> #### [CVE-2007-5342] Permissions, Privileges, and Access Controls
> The default catalina.policy in the JULI logging component in Apache Tomcat 5.5.9 through 5.5.25 and 6.0.0 through 6.0.15 does not restrict certain permissions for web applications, which allows attackers to modify logging configuration options and overwrite arbitrary files, as demonstrated by changing the (1) level, (2) directory, and (3) prefix attributes in the org.apache.juli.FileHandler handler.
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2000-0759</summary>
> #### [CVE-2000-0759] Jakarta Tomcat 3.1 under Apache reveals physical path information when a remote ...
> Jakarta Tomcat 3.1 under Apache reveals physical path information when a remote attacker requests a URL that does not exist, which generates an error message that includes the physical path.
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2010-4312</summary>
> #### [CVE-2010-4312] null
> The default configuration of Apache Tomcat 6.x does not include the HTTPOnly flag in a Set-Cookie header, which makes it easier for remote attackers to hijack a session via script access to a cookie.
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:P/A:P
</details>
<details>
<summary>CVE-2000-0760</summary>
> #### [CVE-2000-0760] The Snoop servlet in Jakarta Tomcat 3.1 and 3.0 under Apache reveals sensitive s...
> The Snoop servlet in Jakarta Tomcat 3.1 and 3.0 under Apache reveals sensitive system information when a remote attacker requests a nonexistent URL with a .snp extension.
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2014-0227</summary>
> #### [CVE-2014-0227] Data Handling
> java/org/apache/coyote/http11/filters/ChunkedInputFilter.java in Apache Tomcat 6.x before 6.0.42, 7.x before 7.0.55, and 8.x before 8.0.9 does not properly handle attempts to continue reading data after an error has occurred, which allows remote attackers to conduct HTTP request smuggling attacks or cause a denial of service (resource consumption) by streaming data with malformed chunked transfer coding.
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:P/A:P
</details>
<details>
<summary>CVE-2016-0763</summary>
> #### [CVE-2016-0763] Permissions, Privileges, and Access Controls
> The setGlobalContext method in org/apache/naming/factory/ResourceLinkFactory.java in Apache Tomcat 7.x before 7.0.68, 8.x before 8.0.31, and 9.x before 9.0.0.M3 does not consider whether ResourceLinkFactory.setGlobalContext callers are authorized, which allows remote authenticated users to bypass intended SecurityManager restrictions and read or write to arbitrary application data, or cause a denial of service (application disruption), via a web application that sets a crafted global context.
>
> **CVSS Score:** 6.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:L
</details>
<details>
<summary>CVE-2019-0221</summary>
> #### [CVE-2019-0221] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> The SSI printenv command in Apache Tomcat 9.0.0.M1 to 9.0.0.17, 8.5.0 to 8.5.39 and 7.0.0 to 7.0.93 echoes user provided data without escaping and is, therefore, vulnerable to XSS. SSI is disabled by default. The printenv command is intended for debugging and is unlikely to be present in a production website.
>
> **CVSS Score:** 6.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:C/C:L/I:L/A:N
</details>
<details>
<summary>CVE-2018-1304</summary>
> #### [CVE-2018-1304] The URL pattern of "" (the empty string) which exactly maps to the context root ...
> The URL pattern of "" (the empty string) which exactly maps to the context root was not correctly handled in Apache Tomcat 9.0.0.M1 to 9.0.4, 8.5.0 to 8.5.27, 8.0.0.RC1 to 8.0.49 and 7.0.0 to 7.0.84 when used as part of a security constraint definition. This caused the constraint to be ignored. It was, therefore, possible for unauthorised users to gain access to web application resources that should have been protected. Only security constraints with a URL pattern of the empty string were affected.
>
> **CVSS Score:** 5.9
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2018-8037</summary>
> #### [CVE-2018-8037] Concurrent Execution using Shared Resource with Improper Synchronization ("Race Condition")
> If an async request was completed by the application at the same time as the container triggered the async timeout, a race condition existed that could result in a user seeing a response intended for a different user. An additional issue was present in the NIO and NIO2 connectors that did not correctly track the closure of the connection when an async request was completed by the application and timed out by the container at the same time. This could also result in a user seeing a response intended for another user. Versions Affected: Apache Tomcat 9.0.0.M9 to 9.0.9 and 8.5.5 to 8.5.31.
>
> **CVSS Score:** 5.9
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2016-0762</summary>
> #### [CVE-2016-0762] Permissions, Privileges, and Access Controls
> The Realm implementations in Apache Tomcat versions 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0.0 to 7.0.70 and 6.0.0 to 6.0.45 did not process the supplied password if the supplied user name did not exist. This made a timing attack possible to determine valid user names. Note that the default configuration includes the LockOutRealm which makes exploitation of this vulnerability harder.
>
> **CVSS Score:** 5.9
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2008-0002</summary>
> #### [CVE-2008-0002] Apache Tomcat 6.0.0 through 6.0.15 processes parameters in the context of the wr...
> Apache Tomcat 6.0.0 through 6.0.15 processes parameters in the context of the wrong request when an exception occurs during parameter processing, which might allow remote attackers to obtain sensitive information, as demonstrated by disconnecting during this processing in order to trigger the exception.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2011-1088</summary>
> #### [CVE-2011-1088] Apache Tomcat 7.x before 7.0.10 does not follow ServletSecurity annotations, whi...
> Apache Tomcat 7.x before 7.0.10 does not follow ServletSecurity annotations, which allows remote attackers to bypass intended access restrictions via HTTP requests to a web application.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2011-1183</summary>
> #### [CVE-2011-1183] Apache Tomcat 7.0.11, when web.xml has no login configuration, does not follow s...
> Apache Tomcat 7.0.11, when web.xml has no login configuration, does not follow security constraints, which allows remote attackers to bypass intended access restrictions via HTTP requests to a meta-data complete web application. NOTE: this vulnerability exists because of an incorrect fix for CVE-2011-1088 and CVE-2011-1419.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2013-4286</summary>
> #### [CVE-2013-4286] Improper Input Validation
> Apache Tomcat before 6.0.39, 7.x before 7.0.47, and 8.x before 8.0.0-RC3, when an HTTP connector or AJP connector is used, does not properly handle certain inconsistent HTTP request headers, which allows remote attackers to trigger incorrect identification of a request's length and conduct request-smuggling attacks via (1) multiple Content-Length headers or (2) a Content-Length header and a "Transfer-Encoding: chunked" header. NOTE: this vulnerability exists because of an incomplete fix for CVE-2005-2090.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2011-1419</summary>
> #### [CVE-2011-1419] Apache Tomcat 7.x before 7.0.11, when web.xml has no security constraints, does ...
> Apache Tomcat 7.x before 7.0.11, when web.xml has no security constraints, does not follow ServletSecurity annotations, which allows remote attackers to bypass intended access restrictions via HTTP requests to a web application. NOTE: this vulnerability exists because of an incomplete fix for CVE-2011-1088.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2009-2693</summary>
> #### [CVE-2009-2693] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Directory traversal vulnerability in Apache Tomcat 5.5.0 through 5.5.28 and 6.0.0 through 6.0.20 allows remote attackers to create or overwrite arbitrary files via a .. (dot dot) in an entry in a WAR file, as demonstrated by a ../../bin/catalina.bat entry.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:P
</details>
<details>
<summary>CVE-2021-33037</summary>
> #### [CVE-2021-33037] Apache Tomcat 10.0.0-M1 to 10.0.6, 9.0.0.M1 to 9.0.46 and 8.5.0 to 8.5.66 did no...
> Apache Tomcat 10.0.0-M1 to 10.0.6, 9.0.0.M1 to 9.0.46 and 8.5.0 to 8.5.66 did not correctly parse the HTTP transfer-encoding request header in some circumstances leading to the possibility to request smuggling when used with a reverse proxy. Specifically: - Tomcat incorrectly ignored the transfer encoding header if the client declared it would only accept an HTTP/1.0 response; - Tomcat honoured the identify encoding; and - Tomcat did not ensure that, if present, the chunked encoding was the final encoding.
>
> **CVSS Score:** 5.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:L/A:N
</details>
<details>
<summary>CVE-2015-5345</summary>
> #### [CVE-2015-5345] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> The Mapper component in Apache Tomcat 6.x before 6.0.45, 7.x before 7.0.68, 8.x before 8.0.30, and 9.x before 9.0.0.M2 processes redirects before considering security constraints and Filters, which allows remote attackers to determine the existence of a directory via a URL that lacks a trailing / (slash) character.
>
> **CVSS Score:** 5.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N
</details>
<details>
<summary>CVE-2017-15706</summary>
> #### [CVE-2017-15706] Improperly Implemented Security Check for Standard
> As part of the fix for bug 61201, the documentation for Apache Tomcat 9.0.0.M22 to 9.0.1, 8.5.16 to 8.5.23, 8.0.45 to 8.0.47 and 7.0.79 to 7.0.82 included an updated description of the search algorithm used by the CGI Servlet to identify which script to execute. The update was not correct. As a result, some scripts may have failed to execute as expected and other scripts may have been executed unexpectedly. Note that the behaviour of the CGI servlet has remained unchanged in this regard. It is only the documentation of the behaviour that was wrong and has been corrected.
>
> **CVSS Score:** 5.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:L/A:N
</details>
<details>
<summary>CVE-2016-6794</summary>
> #### [CVE-2016-6794] Information Exposure
> When a SecurityManager is configured, a web application's ability to read system properties should be controlled by the SecurityManager. In Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0.0 to 7.0.70, 6.0.0 to 6.0.45 the system property replacement feature for configuration files could be used by a malicious web application to bypass the SecurityManager and read system properties that should not be visible.
>
> **CVSS Score:** 5.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N
</details>
<details>
<summary>CVE-2001-0829</summary>
> #### [CVE-2001-0829] A cross-site scripting vulnerability in Apache Tomcat 3.2.1 allows a malicious w...
> A cross-site scripting vulnerability in Apache Tomcat 3.2.1 allows a malicious webmaster to embed Javascript in a request for a .JSP file, which causes the Javascript to be inserted into an error message.
>
> **CVSS Score:** 5.1
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2011-1184</summary>
> #### [CVE-2011-1184] Permissions, Privileges, and Access Controls
> The HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.12 does not have the expected countermeasures against replay attacks, which makes it easier for remote attackers to bypass intended access restrictions by sniffing the network for valid requests, related to lack of checking of nonce (aka server nonce) and nc (aka nonce-count or client nonce count) values.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-2370</summary>
> #### [CVE-2008-2370] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Apache Tomcat 4.1.0 through 4.1.37, 5.5.0 through 5.5.26, and 6.0.0 through 6.0.16, when a RequestDispatcher is used, performs path normalization before removing the query string from the URI, which allows remote attackers to conduct directory traversal attacks and read arbitrary files via a .. (dot dot) in a request parameter.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2011-4858</summary>
> #### [CVE-2011-4858] Resource Management Errors
> Apache Tomcat before 5.5.35, 6.x before 6.0.35, and 7.x before 7.0.23 computes hash values for form parameters without restricting the ability to trigger hash collisions predictably, which allows remote attackers to cause a denial of service (CPU consumption) by sending many crafted parameters.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2012-2733</summary>
> #### [CVE-2012-2733] Improper Input Validation
> java/org/apache/coyote/http11/InternalNioInputBuffer.java in the HTTP NIO connector in Apache Tomcat 6.x before 6.0.36 and 7.x before 7.0.28 does not properly restrict the request-header size, which allows remote attackers to cause a denial of service (memory consumption) via a large amount of header data.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2008-0128</summary>
> #### [CVE-2008-0128] null
> The SingleSignOn Valve (org.apache.catalina.authenticator.SingleSignOn) in Apache Tomcat before 5.5.21 does not set the secure flag for the JSESSIONIDSSO cookie in an https session, which can cause the cookie to be sent in http requests and make it easier for remote attackers to capture this cookie.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-2006</summary>
> #### [CVE-2002-2006] The default installation of Apache Tomcat 4.0 through 4.1 and 3.0 through 3.3.1 ...
> The default installation of Apache Tomcat 4.0 through 4.1 and 3.0 through 3.3.1 allows remote attackers to obtain the installation path and other sensitive system information via the (1) SnoopServlet or (2) TroubleShooter example servlets.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2011-0534</summary>
> #### [CVE-2011-0534] Resource Management Errors
> Apache Tomcat 7.0.0 through 7.0.6 and 6.0.0 through 6.0.30 does not enforce the maxHttpHeaderSize limit for requests involving the NIO HTTP connector, which allows remote attackers to cause a denial of service (OutOfMemoryError) via a crafted request.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2002-1895</summary>
> #### [CVE-2002-1895] The servlet engine in Jakarta Apache Tomcat 3.3 and 4.0.4, when using IIS and th...
> The servlet engine in Jakarta Apache Tomcat 3.3 and 4.0.4, when using IIS and the ajp1.3 connector, allows remote attackers to cause a denial of service (crash) via a large number of HTTP GET requests for an MS-DOS device such as AUX, LPT1, CON, or PRN.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2007-0450</summary>
> #### [CVE-2007-0450] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Directory traversal vulnerability in Apache HTTP Server and Tomcat 5.x before 5.5.22 and 6.x before 6.0.10, when using certain proxy modules (mod_proxy, mod_rewrite, mod_jk), allows remote attackers to read arbitrary files via a .. (dot dot) sequence with combinations of (1) "/" (slash), (2) "\" (backslash), and (3) URL-encoded backslash (%5C) characters in the URL, which are valid separators in Tomcat but not in Apache.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2001-0917</summary>
> #### [CVE-2001-0917] Jakarta Tomcat 4.0.1 allows remote attackers to reveal physical path information...
> Jakarta Tomcat 4.0.1 allows remote attackers to reveal physical path information by requesting a long URL with a .JSP extension.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2005-0808</summary>
> #### [CVE-2005-0808] Apache Tomcat before 5.x allows remote attackers to cause a denial of service (a...
> Apache Tomcat before 5.x allows remote attackers to cause a denial of service (application crash) via a crafted AJP12 packet to TCP port 8007.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2012-0022</summary>
> #### [CVE-2012-0022] Numeric Errors
> Apache Tomcat 5.5.x before 5.5.35, 6.x before 6.0.34, and 7.x before 7.0.23 uses an inefficient approach for handling parameters, which allows remote attackers to cause a denial of service (CPU consumption) via a request that contains many parameters and parameter values, a different vulnerability than CVE-2011-4858.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2000-0672</summary>
> #### [CVE-2000-0672] The default configuration of Jakarta Tomcat does not restrict access to the /adm...
> The default configuration of Jakarta Tomcat does not restrict access to the /admin context, which allows remote attackers to read arbitrary files by directly calling the administrative servlets to add a context for the root directory.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2012-5568</summary>
> #### [CVE-2012-5568] null
> Apache Tomcat through 7.0.x allows remote attackers to cause a denial of service (daemon outage) via partial HTTP requests, as demonstrated by Slowloris.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2011-1475</summary>
> #### [CVE-2011-1475] Improper Input Validation
> The HTTP BIO connector in Apache Tomcat 7.0.x before 7.0.12 does not properly handle HTTP pipelining, which allows remote attackers to read responses intended for other clients in opportunistic circumstances by examining the application data in HTTP packets, related to "a mix-up of responses for requests from different users."
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-2009</summary>
> #### [CVE-2002-2009] Apache Tomcat 4.0.1 allows remote attackers to obtain the web root path via HTTP...
> Apache Tomcat 4.0.1 allows remote attackers to obtain the web root path via HTTP requests for JSP files preceded by (1) +/, (2) >/, (3) </, and (4) %20/, which leaks the pathname in an error message.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2003-0043</summary>
> #### [CVE-2003-0043] Jakarta Tomcat before 3.3.1a, when used with JDK 1.3.1 or earlier, uses trusted ...
> Jakarta Tomcat before 3.3.1a, when used with JDK 1.3.1 or earlier, uses trusted privileges when processing the web.xml file, which could allow remote attackers to read portions of some files through the web.xml file.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2012-5886</summary>
> #### [CVE-2012-5886] Improper Authentication
> The HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.36, 6.x before 6.0.36, and 7.x before 7.0.30 caches information about the authenticated user within the session state, which makes it easier for remote attackers to bypass authentication via vectors related to the session ID.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2014-7810</summary>
> #### [CVE-2014-7810] Improper Access Control
> The Expression Language (EL) implementation in Apache Tomcat 6.x before 6.0.44, 7.x before 7.0.58, and 8.x before 8.0.16 does not properly consider the possibility of an accessible interface implemented by an inaccessible class, which allows attackers to bypass a SecurityManager protection mechanism via a web application that leverages use of incorrect privileges during EL evaluation.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2011-2729</summary>
> #### [CVE-2011-2729] Permissions, Privileges, and Access Controls
> native/unix/native/jsvc-unix.c in jsvc in the Daemon component 1.0.3 through 1.0.6 in Apache Commons, as used in Apache Tomcat 5.5.32 through 5.5.33, 6.0.30 through 6.0.32, and 7.0.x before 7.0.20 on Linux, does not drop capabilities, which allows remote attackers to bypass read permissions for files via a request to an application.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2003-0042</summary>
> #### [CVE-2003-0042] Jakarta Tomcat before 3.3.1a, when used with JDK 1.3.1 or earlier, allows remote...
> Jakarta Tomcat before 3.3.1a, when used with JDK 1.3.1 or earlier, allows remote attackers to list directories even with an index.html or other file present, or obtain unprocessed source code for a JSP file, via a URL containing a null character.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-0935</summary>
> #### [CVE-2002-0935] Apache Tomcat 4.0.3, and possibly other versions before 4.1.3 beta, allows remot...
> Apache Tomcat 4.0.3, and possibly other versions before 4.1.3 beta, allows remote attackers to cause a denial of service (resource exhaustion) via a large number of requests to the server with null characters, which causes the working threads to hang.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2003-0045</summary>
> #### [CVE-2003-0045] Jakarta Tomcat before 3.3.1a on certain Windows systems may allow remote attacke...
> Jakarta Tomcat before 3.3.1a on certain Windows systems may allow remote attackers to cause a denial of service (thread hang and resource consumption) via a request for a JSP page containing an MS-DOS device name, such as aux.jsp.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2007-5333</summary>
> #### [CVE-2007-5333] Information Exposure
> Apache Tomcat 6.0.0 through 6.0.14, 5.5.0 through 5.5.25, and 4.1.0 through 4.1.36 does not properly handle (1) double quote (") characters or (2) %5C (encoded backslash) sequences in a cookie value, which might cause sensitive information such as session IDs to be leaked to remote attackers and enable session hijacking attacks. NOTE: this issue exists because of an incomplete fix for CVE-2007-3385.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2014-0095</summary>
> #### [CVE-2014-0095] Improper Input Validation
> java/org/apache/coyote/ajp/AbstractAjpProcessor.java in Apache Tomcat 8.x before 8.0.4 allows remote attackers to cause a denial of service (thread consumption) by using a "Content-Length: 0" AJP request to trigger a hang in request processing.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2012-3544</summary>
> #### [CVE-2012-3544] Improper Input Validation
> Apache Tomcat 6.x before 6.0.37 and 7.x before 7.0.30 does not properly handle chunk extensions in chunked transfer coding, which allows remote attackers to cause a denial of service by streaming data.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2005-4703</summary>
> #### [CVE-2005-4703] Apache Tomcat 4.0.3, when running on Windows, allows remote attackers to obtain ...
> Apache Tomcat 4.0.3, when running on Windows, allows remote attackers to obtain sensitive information via a request for a file that contains an MS-DOS device name such as lpt9, which leaks the pathname in an error message, as demonstrated by lpt9.xtp using Nikto.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-0936</summary>
> #### [CVE-2002-0936] The Java Server Pages (JSP) engine in Tomcat allows web page owners to cause a d...
> The Java Server Pages (JSP) engine in Tomcat allows web page owners to cause a denial of service (engine crash) on the web server via a JSP page that calls WPrinterJob().pageSetup(null,null).
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2006-3835</summary>
> #### [CVE-2006-3835] Apache Tomcat 5 before 5.5.17 allows remote attackers to list directories via a ...
> Apache Tomcat 5 before 5.5.17 allows remote attackers to list directories via a semicolon (;) preceding a filename with a mapped extension, as demonstrated by URLs ending with /;index.jsp and /;help.do.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-2007</summary>
> #### [CVE-2002-2007] The default installations of Apache Tomcat 3.2.3 and 3.2.4 allows remote attacke...
> The default installations of Apache Tomcat 3.2.3 and 3.2.4 allows remote attackers to obtain sensitive system information such as directory listings and web root path, via erroneous HTTP requests for Java Server Pages (JSP) in the (1) test/jsp, (2) samples/jsp and (3) examples/jsp directories, or the (4) test/realPath.jsp servlet, which leaks pathnames in error messages.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-2008</summary>
> #### [CVE-2002-2008] Apache Tomcat 4.0.3 for Windows allows remote attackers to obtain the web root p...
> Apache Tomcat 4.0.3 for Windows allows remote attackers to obtain the web root path via an HTTP request for a resource that does not exist, such as lpt9, which leaks the information in an error message.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2001-0590</summary>
> #### [CVE-2001-0590] Apache Software Foundation Tomcat Servlet prior to 3.2.2 allows a remote attacke...
> Apache Software Foundation Tomcat Servlet prior to 3.2.2 allows a remote attacker to read the source code to arbitrary 'jsp' files via a malformed URL request which does not end with an HTTP protocol specification (i.e. HTTP/1.0).
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2009-0033</summary>
> #### [CVE-2009-0033] Improper Input Validation
> Apache Tomcat 4.1.0 through 4.1.39, 5.5.0 through 5.5.27, and 6.0.0 through 6.0.18, when the Java AJP connector and mod_jk load balancing are used, allows remote attackers to cause a denial of service (application outage) via a crafted request with invalid headers, related to temporary blocking of connectors that have encountered errors, as demonstrated by an error involving a malformed HTTP Host header.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2012-5887</summary>
> #### [CVE-2012-5887] Improper Authentication
> The HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.36, 6.x before 6.0.36, and 7.x before 7.0.30 does not properly check for stale nonce values in conjunction with enforcement of proper credentials, which makes it easier for remote attackers to bypass intended access restrictions by sniffing the network for valid requests.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2002-1148</summary>
> #### [CVE-2002-1148] The default servlet (org.apache.catalina.servlets.DefaultServlet) in Tomcat 4.0....
> The default servlet (org.apache.catalina.servlets.DefaultServlet) in Tomcat 4.0.4 and 4.1.10 and earlier allows remote attackers to read source code for server files via a direct request to the servlet.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2003-0866</summary>
> #### [CVE-2003-0866] The Catalina org.apache.catalina.connector.http package in Tomcat 4.0.x up to 4....
> The Catalina org.apache.catalina.connector.http package in Tomcat 4.0.x up to 4.0.3 allows remote attackers to cause a denial of service via several requests that do not follow the HTTP protocol, which causes Tomcat to reject later requests.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2011-3375</summary>
> #### [CVE-2011-3375] Information Exposure
> Apache Tomcat 6.0.30 through 6.0.33 and 7.x before 7.0.22 does not properly perform certain caching and recycling operations involving request objects, which allows remote attackers to obtain unintended read access to IP address and HTTP header information in opportunistic circumstances by reading TCP data.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2014-0075</summary>
> #### [CVE-2014-0075] Numeric Errors
> Integer overflow in the parseChunkHeader function in java/org/apache/coyote/http11/filters/ChunkedInputFilter.java in Apache Tomcat before 6.0.40, 7.x before 7.0.53, and 8.x before 8.0.4 allows remote attackers to cause a denial of service (resource consumption) via a malformed chunk size in chunked transfer coding of a request during the streaming of data.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2011-5062</summary>
> #### [CVE-2011-5062] Permissions, Privileges, and Access Controls
> The HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.12 does not check qop values, which might allow remote attackers to bypass intended integrity-protection requirements via a qop=auth value, a different vulnerability than CVE-2011-1184.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2000-1210</summary>
> #### [CVE-2000-1210] Directory traversal vulnerability in source.jsp of Apache Tomcat before 3.1 allo...
> Directory traversal vulnerability in source.jsp of Apache Tomcat before 3.1 allows remote attackers to read arbitrary files via a .. (dot dot) in the argument to source.jsp.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-5515</summary>
> #### [CVE-2008-5515] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Apache Tomcat 4.1.0 through 4.1.39, 5.5.0 through 5.5.27, 6.0.0 through 6.0.18, and possibly earlier versions normalizes the target pathname before filtering the query string when using the RequestDispatcher method, which allows remote attackers to bypass intended access restrictions and conduct directory traversal attacks via .. (dot dot) sequences and the WEB-INF directory in a Request.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2005-3510</summary>
> #### [CVE-2005-3510] Apache Tomcat 5.5.0 to 5.5.11 allows remote attackers to cause a denial of servi...
> Apache Tomcat 5.5.0 to 5.5.11 allows remote attackers to cause a denial of service (CPU consumption) via a large number of simultaneous requests to list a web directory that has a large number of files.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2012-5885</summary>
> #### [CVE-2012-5885] Permissions, Privileges, and Access Controls
> The replay-countermeasure functionality in the HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.36, 6.x before 6.0.36, and 7.x before 7.0.30 tracks cnonce (aka client nonce) values instead of nonce (aka server nonce) and nc (aka nonce-count) values, which makes it easier for remote attackers to bypass intended access restrictions by sniffing the network for valid requests, a different vulnerability than CVE-2011-1184.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2019-17569</summary>
> #### [CVE-2019-17569] The refactoring present in Apache Tomcat 9.0.28 to 9.0.30, 8.5.48 to 8.5.50 and ...
> The refactoring present in Apache Tomcat 9.0.28 to 9.0.30, 8.5.48 to 8.5.50 and 7.0.98 to 7.0.99 introduced a regression. The result of the regression was that invalid Transfer-Encoding headers were incorrectly processed leading to a possibility of HTTP Request Smuggling if Tomcat was located behind a reverse proxy that incorrectly handled the invalid Transfer-Encoding header in a particular manner. Such a reverse proxy is considered unlikely.
>
> **CVSS Score:** 4.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:L/A:N
</details>
<details>
<summary>CVE-2020-1935</summary>
> #### [CVE-2020-1935] In Apache Tomcat 9.0.0.M1 to 9.0.30, 8.5.0 to 8.5.50 and 7.0.0 to 7.0.99 the HTT...
> In Apache Tomcat 9.0.0.M1 to 9.0.30, 8.5.0 to 8.5.50 and 7.0.0 to 7.0.99 the HTTP header parsing code used an approach to end-of-line parsing that allowed some invalid HTTP headers to be parsed as valid. This led to a possibility of HTTP Request Smuggling if Tomcat was located behind a reverse proxy that incorrectly handled the invalid Transfer-Encoding header in a particular manner. Such a reverse proxy is considered unlikely.
>
> **CVSS Score:** 4.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:L/A:N
</details>
<details>
<summary>CVE-2011-2481</summary>
> #### [CVE-2011-2481] Apache Tomcat 7.0.x before 7.0.17 permits web applications to replace an XML par...
> Apache Tomcat 7.0.x before 7.0.17 permits web applications to replace an XML parser used for other web applications, which allows local users to read or modify the (1) web.xml, (2) context.xml, or (3) tld files of arbitrary web applications via a crafted application that is loaded earlier than the target application. NOTE: this vulnerability exists because of a CVE-2009-0783 regression.
>
> **CVSS Score:** 4.6
>
> **CVSS Vector:** AV:L/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2011-3376</summary>
> #### [CVE-2011-3376] Permissions, Privileges, and Access Controls
> org/apache/catalina/core/DefaultInstanceManager.java in Apache Tomcat 7.x before 7.0.22 does not properly restrict ContainerServlets in the Manager application, which allows local users to gain privileges by using an untrusted web application to access the Manager application's functionality.
>
> **CVSS Score:** 4.4
>
> **CVSS Vector:** AV:L/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2011-2526</summary>
> #### [CVE-2011-2526] Improper Input Validation
> Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.19, when sendfile is enabled for the HTTP APR or HTTP NIO connector, does not validate certain request attributes, which allows local users to bypass intended file access restrictions or cause a denial of service (infinite loop or JVM crash) by leveraging an untrusted web application.
>
> **CVSS Score:** 4.4
>
> **CVSS Vector:** AV:L/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2011-5064</summary>
> #### [CVE-2011-5064] Cryptographic Issues
> DigestAuthenticator.java in the HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.12 uses Catalina as the hard-coded server secret (aka private key), which makes it easier for remote attackers to bypass cryptographic protection mechanisms by leveraging knowledge of this string, a different vulnerability than CVE-2011-1184.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-1947</summary>
> #### [CVE-2008-1947] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in Apache Tomcat 5.5.9 through 5.5.26 and 6.0.0 through 6.0.16 allows remote attackers to inject arbitrary web script or HTML via the name parameter (aka the hostname attribute) to host-manager/html/add.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2016-0706</summary>
> #### [CVE-2016-0706] Information Exposure
> Apache Tomcat 6.x before 6.0.45, 7.x before 7.0.68, 8.x before 8.0.31, and 9.x before 9.0.0.M2 does not place org.apache.catalina.manager.StatusManagerServlet on the org/apache/catalina/core/RestrictedServlets.properties list, which allows remote authenticated users to bypass intended SecurityManager restrictions and read arbitrary HTTP requests, and consequently discover session ID values, via a crafted web application.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N
</details>
<details>
<summary>CVE-2014-0119</summary>
> #### [CVE-2014-0119] Permissions, Privileges, and Access Controls
> Apache Tomcat before 6.0.40, 7.x before 7.0.54, and 8.x before 8.0.6 does not properly constrain the class loader that accesses the XML parser used with an XSLT stylesheet, which allows remote attackers to (1) read arbitrary files via a crafted web application that provides an XML external entity declaration in conjunction with an entity reference, related to an XML External Entity (XXE) issue, or (2) read files associated with different web applications on a single Tomcat instance via a crafted web application.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2006-7195</summary>
> #### [CVE-2006-7195] Cross-site scripting (XSS) vulnerability in implicit-objects.jsp in Apache Tomca...
> Cross-site scripting (XSS) vulnerability in implicit-objects.jsp in Apache Tomcat 5.0.0 through 5.0.30 and 5.5.0 through 5.5.17 allows remote attackers to inject arbitrary web script or HTML via certain header values.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2013-4322</summary>
> #### [CVE-2013-4322] Improper Input Validation
> Apache Tomcat before 6.0.39, 7.x before 7.0.50, and 8.x before 8.0.0-RC10 processes chunked transfer coding without properly handling (1) a large total amount of chunked data or (2) whitespace characters in an HTTP header value within a trailer field, which allows remote attackers to cause a denial of service by streaming data. NOTE: this vulnerability exists because of an incomplete fix for CVE-2012-3544.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2011-1582</summary>
> #### [CVE-2011-1582] Permissions, Privileges, and Access Controls
> Apache Tomcat 7.0.12 and 7.0.13 processes the first request to a servlet without following security constraints that have been configured through annotations, which allows remote attackers to bypass intended access restrictions via HTTP requests. NOTE: this vulnerability exists because of an incomplete fix for CVE-2011-1088, CVE-2011-1183, and CVE-2011-1419.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2010-4172</summary>
> #### [CVE-2010-4172] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Multiple cross-site scripting (XSS) vulnerabilities in the Manager application in Apache Tomcat 6.0.12 through 6.0.29 and 7.0.0 through 7.0.4 allow remote attackers to inject arbitrary web script or HTML via the (1) orderBy or (2) sort parameter to sessionsList.jsp, or unspecified input to (3) sessionDetail.jsp or (4) java/org/apache/catalina/manager/JspHelper.java, related to use of untrusted web applications.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-2449</summary>
> #### [CVE-2007-2449] Multiple cross-site scripting (XSS) vulnerabilities in certain JSP files in the ...
> Multiple cross-site scripting (XSS) vulnerabilities in certain JSP files in the examples web application in Apache Tomcat 4.0.0 through 4.0.6, 4.1.0 through 4.1.36, 5.0.0 through 5.0.30, 5.5.0 through 5.5.24, and 6.0.0 through 6.0.13 allow remote attackers to inject arbitrary web script or HTML via the portion of the URI after the ';' character, as demonstrated by a URI containing a "snp/snoop.jsp;" sequence.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2012-4431</summary>
> #### [CVE-2012-4431] Permissions, Privileges, and Access Controls
> org/apache/catalina/filters/CsrfPreventionFilter.java in Apache Tomcat 6.x before 6.0.36 and 7.x before 7.0.32 allows remote attackers to bypass the cross-site request forgery (CSRF) protection mechanism via a request that lacks a session identifier.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-3385</summary>
> #### [CVE-2007-3385] Information Exposure
> Apache Tomcat 6.0.0 to 6.0.13, 5.5.0 to 5.5.24, 5.0.0 to 5.0.30, 4.1.0 to 4.1.36, and 3.3 to 3.3.2 does not properly handle the \" character sequence in a cookie value, which might cause sensitive information such as session IDs to be leaked to remote attackers and enable session hijacking attacks.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2006-7196</summary>
> #### [CVE-2006-7196] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in the calendar application example in Apache Tomcat 4.0.0 through 4.0.6, 4.1.0 through 4.1.31, 5.0.0 through 5.0.30, and 5.5.0 through 5.5.15 allows remote attackers to inject arbitrary web script or HTML via the time parameter to cal2.jsp and possibly unspecified other vectors. NOTE: this may be related to CVE-2006-0254.1.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-3384</summary>
> #### [CVE-2007-3384] Multiple cross-site scripting (XSS) vulnerabilities in examples/servlet/CookieEx...
> Multiple cross-site scripting (XSS) vulnerabilities in examples/servlet/CookieExample in Apache Tomcat 3.3 through 3.3.2 allow remote attackers to inject arbitrary web script or HTML via the (1) Name or (2) Value field, related to error messages.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2011-5063</summary>
> #### [CVE-2011-5063] Improper Authentication
> The HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.12 does not check realm values, which might allow remote attackers to bypass intended access restrictions by leveraging the availability of a protection space with weaker authentication or authorization requirements, a different vulnerability than CVE-2011-1184.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2014-0096</summary>
> #### [CVE-2014-0096] Permissions, Privileges, and Access Controls
> java/org/apache/catalina/servlets/DefaultServlet.java in the default servlet in Apache Tomcat before 6.0.40, 7.x before 7.0.53, and 8.x before 8.0.4 does not properly restrict XSLT stylesheets, which allows remote attackers to bypass security-manager restrictions and read arbitrary files via a crafted web application that provides an XML external entity declaration in conjunction with an entity reference, related to an XML External Entity (XXE) issue.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-1232</summary>
> #### [CVE-2008-1232] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in Apache Tomcat 4.1.0 through 4.1.37, 5.5.0 through 5.5.26, and 6.0.0 through 6.0.16 allows remote attackers to inject arbitrary web script or HTML via a crafted string that is used in the message argument to the HttpServletResponse.sendError method.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-3382</summary>
> #### [CVE-2007-3382] Information Exposure
> Apache Tomcat 6.0.0 to 6.0.13, 5.5.0 to 5.5.24, 5.0.0 to 5.0.30, 4.1.0 to 4.1.36, and 3.3 to 3.3.2 treats single quotes ("'") as delimiters in cookies, which might cause sensitive information such as session IDs to be leaked and allow remote attackers to conduct session hijacking attacks.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2017-7674</summary>
> #### [CVE-2017-7674] Insufficient Verification of Data Authenticity
> The CORS Filter in Apache Tomcat 9.0.0.M1 to 9.0.0.M21, 8.5.0 to 8.5.15, 8.0.0.RC1 to 8.0.44 and 7.0.41 to 7.0.78 did not add an HTTP Vary header indicating that the response varies depending on Origin. This permitted client and server side cache poisoning in some circumstances.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:L/A:N
</details>
<details>
<summary>CVE-2007-4724</summary>
> #### [CVE-2007-4724] Cross-Site Request Forgery (CSRF)
> Cross-site request forgery (CSRF) vulnerability in cal2.jsp in the calendar examples application in Apache Tomcat 4.1.31 allows remote attackers to add events as arbitrary users via the time and description parameters.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2015-5174</summary>
> #### [CVE-2015-5174] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Directory traversal vulnerability in RequestUtil.java in Apache Tomcat 6.x before 6.0.45, 7.x before 7.0.65, and 8.x before 8.0.27 allows remote authenticated users to bypass intended SecurityManager restrictions and list a parent directory via a /.. (slash dot dot) in a pathname used by a web application in a getResource, getResourceAsStream, or getResourcePaths call, as demonstrated by the $CATALINA_BASE/webapps directory.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N
</details>
<details>
<summary>CVE-2009-2901</summary>
> #### [CVE-2009-2901] Permissions, Privileges, and Access Controls
> The autodeployment process in Apache Tomcat 5.5.0 through 5.5.28 and 6.0.0 through 6.0.20, when autoDeploy is enabled, deploys appBase files that remain from a failed undeploy, which might allow remote attackers to bypass intended authentication requirements via HTTP requests.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2007-6286</summary>
> #### [CVE-2007-6286] Apache Tomcat 5.5.11 through 5.5.25 and 6.0.0 through 6.0.15, when the native AP...
> Apache Tomcat 5.5.11 through 5.5.25 and 6.0.0 through 6.0.15, when the native APR connector is used, does not properly handle an empty request to the SSL port, which allows remote attackers to trigger handling of "a duplicate copy of one of the recent requests," as demonstrated by using netcat to send the empty request.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2014-0033</summary>
> #### [CVE-2014-0033] Improper Input Validation
> org/apache/catalina/connector/CoyoteAdapter.java in Apache Tomcat 6.0.33 through 6.0.37 does not consider the disableURLRewriting setting when handling a session ID in a URL, which allows remote attackers to conduct session fixation attacks via a crafted URL.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2009-2902</summary>
> #### [CVE-2009-2902] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Directory traversal vulnerability in Apache Tomcat 5.5.0 through 5.5.28 and 6.0.0 through 6.0.20 allows remote attackers to delete work-directory files via directory traversal sequences in a WAR filename, as demonstrated by the ...war filename.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2009-2696</summary>
> #### [CVE-2009-2696] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in jsp/cal/cal2.jsp in the calendar application in the examples web application in Apache Tomcat on Red Hat Enterprise Linux 5, Desktop Workstation 5, and Linux Desktop 5 allows remote attackers to inject arbitrary web script or HTML via the time parameter, related to "invalid HTML." NOTE: this is due to a missing fix for CVE-2009-0781.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2005-2090</summary>
> #### [CVE-2005-2090] Jakarta Tomcat 5.0.19 (Coyote/1.1) and Tomcat 4.1.24 (Coyote/1.0) allows remote ...
> Jakarta Tomcat 5.0.19 (Coyote/1.1) and Tomcat 4.1.24 (Coyote/1.0) allows remote attackers to poison the web cache, bypass web application firewall protection, and conduct XSS attacks via an HTTP request with both a "Transfer-Encoding: chunked" header and a Content-Length header, which causes Tomcat to incorrectly handle and forward the body of the request in a way that causes the receiving server to process it as a separate HTTP request, aka "HTTP Request Smuggling."
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2005-4838</summary>
> #### [CVE-2005-4838] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Multiple cross-site scripting (XSS) vulnerabilities in the example web applications for Jakarta Tomcat 5.5.6 and earlier allow remote attackers to inject arbitrary web script or HTML via (1) el/functions.jsp, (2) el/implicit-objects.jsp, and (3) jspx/textRotate.jspx in examples/jsp2/, as demonstrated via script in a request to snp/snoop.jsp. NOTE: other XSS issues in the manager were simultaneously reported, but these require admin access and do not cross privilege boundaries.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2008-3271</summary>
> #### [CVE-2008-3271] Permissions, Privileges, and Access Controls
> Apache Tomcat 5.5.0 and 4.1.0 through 4.1.31 allows remote attackers to bypass an IP address restriction and obtain sensitive information via a request that is processed concurrently with another request but in a different thread, leading to an instance-variable overwrite associated with a "synchronization problem" and lack of thread safety, and related to RemoteFilterValve, RemoteAddrValve, and RemoteHostValve.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2020-13943</summary>
> #### [CVE-2020-13943] If an HTTP/2 client connecting to Apache Tomcat 10.0.0-M1 to 10.0.0-M7, 9.0.0.M1...
> If an HTTP/2 client connecting to Apache Tomcat 10.0.0-M1 to 10.0.0-M7, 9.0.0.M1 to 9.0.37 or 8.5.0 to 8.5.57 exceeded the agreed maximum number of concurrent streams for a connection (in violation of the HTTP/2 protocol), it was possible that a subsequent request made on that connection could contain HTTP headers - including HTTP/2 pseudo headers - from a previous request rather than the intended headers. This could lead to users seeing responses for unexpected resources.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N
</details>
<details>
<summary>CVE-2011-0013</summary>
> #### [CVE-2011-0013] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Multiple cross-site scripting (XSS) vulnerabilities in the HTML Manager Interface in Apache Tomcat 5.5 before 5.5.32, 6.0 before 6.0.30, and 7.0 before 7.0.6 allow remote attackers to inject arbitrary web script or HTML, as demonstrated via the display-name tag.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-3383</summary>
> #### [CVE-2007-3383] Cross-site scripting (XSS) vulnerability in SendMailServlet in the examples web ...
> Cross-site scripting (XSS) vulnerability in SendMailServlet in the examples web application (examples/jsp/mail/sendmail.jsp) in Apache Tomcat 4.0.0 through 4.0.6 and 4.1.0 through 4.1.36 allows remote attackers to inject arbitrary web script or HTML via the From field and possibly other fields, related to generation of error messages.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2009-0580</summary>
> #### [CVE-2009-0580] Information Exposure
> Apache Tomcat 4.1.0 through 4.1.39, 5.5.0 through 5.5.27, and 6.0.0 through 6.0.18, when FORM authentication is used, allows remote attackers to enumerate valid usernames via requests to /j_security_check with malformed URL encoding of passwords, related to improper error checking in the (1) MemoryRealm, (2) DataSourceRealm, and (3) JDBCRealm authentication realms, as demonstrated by a % (percent) value for the j_password parameter.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2012-3546</summary>
> #### [CVE-2012-3546] Permissions, Privileges, and Access Controls
> org/apache/catalina/realm/RealmBase.java in Apache Tomcat 6.x before 6.0.36 and 7.x before 7.0.30, when FORM authentication is used, allows remote attackers to bypass security-constraint checks by leveraging a previous setUserPrincipal call and then placing /j_security_check at the end of a URI.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2013-4590</summary>
> #### [CVE-2013-4590] Information Exposure
> Apache Tomcat before 6.0.39, 7.x before 7.0.50, and 8.x before 8.0.0-RC10 allows attackers to obtain "Tomcat internals" information by leveraging the presence of an untrusted web application with a context.xml, web.xml, *.jspx, *.tagx, or *.tld XML document containing an external entity declaration in conjunction with an entity reference, related to an XML External Entity (XXE) issue.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2014-0099</summary>
> #### [CVE-2014-0099] Numeric Errors
> Integer overflow in java/org/apache/tomcat/util/buf/Ascii.java in Apache Tomcat before 6.0.40, 7.x before 7.0.53, and 8.x before 8.0.4, when operated behind a reverse proxy, allows remote attackers to conduct HTTP request smuggling attacks via a crafted Content-Length HTTP header.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2018-11784</summary>
> #### [CVE-2018-11784] URL Redirection to Untrusted Site ("Open Redirect")
> When the default servlet in Apache Tomcat versions 9.0.0.M1 to 9.0.11, 8.5.0 to 8.5.33 and 7.0.23 to 7.0.90 returned a redirect to a directory (e.g. redirecting to '/foo/' when the user requested '/foo') a specially crafted URL could be used to cause the redirect to be generated to any URI of the attackers choice.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:L/A:N
</details>
<details>
<summary>CVE-2007-3386</summary>
> #### [CVE-2007-3386] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in the Host Manager Servlet for Apache Tomcat 6.0.0 to 6.0.13 and 5.5.0 to 5.5.24 allows remote attackers to inject arbitrary HTML and web script via crafted requests, as demonstrated using the aliases parameter to an html/add action.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2009-0781</summary>
> #### [CVE-2009-0781] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in jsp/cal/cal2.jsp in the calendar application in the examples web application in Apache Tomcat 4.1.0 through 4.1.39, 5.5.0 through 5.5.27, and 6.0.0 through 6.0.18 allows remote attackers to inject arbitrary web script or HTML via the time parameter, related to "invalid HTML."
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-1355</summary>
> #### [CVE-2007-1355] Multiple cross-site scripting (XSS) vulnerabilities in the appdev/sample/web/hel...
> Multiple cross-site scripting (XSS) vulnerabilities in the appdev/sample/web/hello.jsp example application in Tomcat 4.0.0 through 4.0.6, 4.1.0 through 4.1.36, 5.0.0 through 5.0.30, 5.5.0 through 5.5.23, and 6.0.0 through 6.0.10 allow remote attackers to inject arbitrary web script or HTML via the test parameter and unspecified vectors.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2008-2938</summary>
> #### [CVE-2008-2938] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Directory traversal vulnerability in Apache Tomcat 4.1.0 through 4.1.37, 5.5.0 through 5.5.26, and 6.0.0 through 6.0.16, when allowLinking and UTF-8 are enabled, allows remote attackers to read arbitrary files via encoded directory traversal sequences in the URI, a different vulnerability than CVE-2008-2370. NOTE: versions earlier than 6.0.18 were reported affected, but the vendor advisory lists 6.0.16 as the last affected version.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2009-0783</summary>
> #### [CVE-2009-0783] Information Exposure
> Apache Tomcat 4.1.0 through 4.1.39, 5.5.0 through 5.5.27, and 6.0.0 through 6.0.18 permits web applications to replace an XML parser used for other web applications, which allows local users to read or modify the (1) web.xml, (2) context.xml, or (3) tld files of arbitrary web applications via a crafted application that is loaded earlier than the target application.
>
> **CVSS Score:** 4.2
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:L/PR:H/UI:N/S:U/C:L/I:L/A:L
</details>
</ul>
</details>
<details>
<summary><b>MODERATE Vulnerabilities (13)</b></summary><br/>
<ul>
<details>
<summary>CVE-2007-5461</summary>
> #### [CVE-2007-5461] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Absolute path traversal vulnerability in Apache Tomcat 4.0.0 through 4.0.6, 4.1.0, 5.0.0, 5.5.0 through 5.5.25, and 6.0.0 through 6.0.14, under certain configurations, allows remote authenticated users to read arbitrary files via a WebDAV write request that specifies an entity with a SYSTEM tag.
>
> **CVSS Score:** 3.5
>
> **CVSS Vector:** AV:N/AC:M/Au:S/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2007-2450</summary>
> #### [CVE-2007-2450] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Multiple cross-site scripting (XSS) vulnerabilities in the (1) Manager and (2) Host Manager web applications in Apache Tomcat 4.0.0 through 4.0.6, 4.1.0 through 4.1.36, 5.0.0 through 5.0.30, 5.5.0 through 5.5.24, and 6.0.0 through 6.0.13 allow remote authenticated users to inject arbitrary web script or HTML via a parameter name to manager/html/upload, and other unspecified vectors.
>
> **CVSS Score:** 3.5
>
> **CVSS Vector:** AV:N/AC:M/Au:S/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2005-3164</summary>
> #### [CVE-2005-3164] Information Exposure
> The AJP connector in Apache Tomcat 4.0.1 through 4.0.6 and 4.1.0 through 4.1.36, as used in Hitachi Cosminexus Application Server and standalone, does not properly handle when a connection is broken before request body data is sent in a POST request, which can lead to an information leak when "unsuitable request body data" is used for a different request, possibly related to Java Servlet pages.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-5519</summary>
> #### [CVE-2008-5519] Information Exposure
> The JK Connector (aka mod_jk) 1.2.0 through 1.2.26 in Apache Tomcat allows remote attackers to obtain sensitive information via an arbitrary request from an HTTP client, in opportunistic circumstances involving (1) a request from a different client that included a Content-Length header but no POST data or (2) a rapid series of requests, related to noncompliance with the AJP protocol's requirements for requests containing Content-Length headers.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2007-1358</summary>
> #### [CVE-2007-1358] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in certain applications using Apache Tomcat 4.0.0 through 4.0.6 and 4.1.0 through 4.1.34 allows remote attackers to inject arbitrary web script or HTML via crafted "Accept-Language headers that do not conform to RFC 2616".
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2013-2071</summary>
> #### [CVE-2013-2071] Information Exposure
> java/org/apache/catalina/core/AsyncContextImpl.java in Apache Tomcat 7.x before 7.0.40 does not properly handle the throwing of a RuntimeException in an AsyncListener in an application, which allows context-dependent attackers to obtain sensitive request information intended for other applications in opportunistic circumstances via an application that records the requests that it processes.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-4308</summary>
> #### [CVE-2008-4308] Information Exposure
> The doRead method in Apache Tomcat 4.1.32 through 4.1.34 and 5.5.10 through 5.5.20 does not return a -1 to indicate when a certain error condition has occurred, which can cause Tomcat to send POST content from one request to a different request.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2010-1157</summary>
> #### [CVE-2010-1157] Information Exposure
> Apache Tomcat 5.5.0 through 5.5.29 and 6.0.0 through 6.0.26 might allow remote attackers to discover the server's hostname or IP address by sending a request for a resource that requires (1) BASIC or (2) DIGEST authentication, and then reading the realm field in the WWW-Authenticate header in the reply.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2012-4534</summary>
> #### [CVE-2012-4534] Resource Management Errors
> org/apache/tomcat/util/net/NioEndpoint.java in Apache Tomcat 6.x before 6.0.36 and 7.x before 7.0.28, when the NIO connector is used in conjunction with sendfile and HTTPS, allows remote attackers to cause a denial of service (infinite loop) by terminating the connection during the reading of a response.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2007-1858</summary>
> #### [CVE-2007-1858] The default SSL cipher configuration in Apache Tomcat 4.1.28 through 4.1.31, 5.0...
> The default SSL cipher configuration in Apache Tomcat 4.1.28 through 4.1.31, 5.0.0 through 5.0.30, and 5.5.0 through 5.5.17 uses certain insecure ciphers, including the anonymous cipher, which allows remote attackers to obtain sensitive information or have other, unspecified impacts.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2013-0346</summary>
> #### [CVE-2013-0346] Permissions, Privileges, and Access Controls
> ** DISPUTED ** Apache Tomcat 7.x uses world-readable permissions for the log directory and its files, which might allow local users to obtain sensitive information by reading a file. NOTE: One Tomcat distributor has stated "The tomcat log directory does not contain any sensitive information."
>
> **CVSS Score:** 2.1
>
> **CVSS Vector:** AV:L/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2011-2204</summary>
> #### [CVE-2011-2204] Information Exposure
> Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.17, when the MemoryUserDatabase is used, creates log entries containing passwords upon encountering errors in JMX user creation, which allows local users to obtain sensitive information by reading a log file.
>
> **CVSS Score:** 1.9
>
> **CVSS Vector:** AV:L/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2010-3718</summary>
> #### [CVE-2010-3718] Apache Tomcat 7.0.0 through 7.0.3, 6.0.x, and 5.5.x, when running within a Secur...
> Apache Tomcat 7.0.0 through 7.0.3, 6.0.x, and 5.5.x, when running within a SecurityManager, does not make the ServletContext attribute read-only, which allows local web applications to read or write files outside of the intended working directory, as demonstrated using a directory traversal attack.
>
> **CVSS Score:** 1.2
>
> **CVSS Vector:** AV:L/AC:H/Au:N/C:N/I:P/A:N
</details>
</ul>
</details>
</ul>
</details>
</ul>
</details>
(at-me [in a reply](https://help.sonatype.com/lift/talking-to-lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-925944356
FYI to all, due to the discussion, the IoTDB latest version(0.12.2) can't support fuzzy query as expected(bug). Although it is fixed in their main branch, we don't have an image to test.
The reason of e2e passed, it's we don't test it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713745255
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,54 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <iotdb-session.version>0.12.2</iotdb-session.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <version>${iotdb-session.version}</version>
Review comment:
```suggestion
```
iotdb client version has been managed by bom.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r742905120
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,17 +61,26 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.13.0-SNAPSHOT.jar
+iotdb-thrift-0.13.0-SNAPSHOT.jar
+istack-commons-runtime-4.0.0.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
jackson-databind-2.12.2.jar
jackson-datatype-guava-2.9.10.jar
jackson-datatype-jdk8-2.9.10.jar
jackson-module-afterburner-2.12.2.jar
jackson-module-kotlin-2.8.8.jar
+jakarta.activation-2.0.0.jar
+jakarta.xml.bind-api-3.0.0.jar
Review comment:
`snapshot` is not always using the local repository, it just has an expired time.
I did local check again with your recommend `-U`, still get the same result.
I checked my iotdb dependencies, they are the latest.
My question is not about IoTDB session version, I was asking about `jakata*`, which CI says they don't exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r742876708
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,17 +61,26 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.13.0-SNAPSHOT.jar
+iotdb-thrift-0.13.0-SNAPSHOT.jar
+istack-commons-runtime-4.0.0.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
jackson-databind-2.12.2.jar
jackson-datatype-guava-2.9.10.jar
jackson-datatype-jdk8-2.9.10.jar
jackson-module-afterburner-2.12.2.jar
jackson-module-kotlin-2.8.8.jar
+jakarta.activation-2.0.0.jar
+jakarta.xml.bind-api-3.0.0.jar
Review comment:
I think it is your local cache that cause this issue, because when maven downloads a SNAPSHOT package it will firstly check whether it exists in your local repository (typically `~/.m2/repository`), and if exists it will be used directly, but SNAPSHOT can be overrode in remote repository so you might be using an old revision, as I check the latest revision of this iotdb-session SNAPSHOT is `iotdb-session-0.13.0-20211102.115555-1.jar Tue Nov 02 14:07:09 UTC 2021 67297` (the same revision GitHub Action is using) so you might want to update the local SNAPSHOT and repackage to see the changes, generally adding `-U` will do the trick (`./mvnw -U ....`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r744257306
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,17 +61,26 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.13.0-SNAPSHOT.jar
+iotdb-thrift-0.13.0-SNAPSHOT.jar
+istack-commons-runtime-4.0.0.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
jackson-databind-2.12.2.jar
jackson-datatype-guava-2.9.10.jar
jackson-datatype-jdk8-2.9.10.jar
jackson-module-afterburner-2.12.2.jar
jackson-module-kotlin-2.8.8.jar
+jakarta.activation-2.0.0.jar
+jakarta.xml.bind-api-3.0.0.jar
Review comment:
OK, it seems need to delete some local jars. Anyway, this would not be an issue after official release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755163388
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBBrowserLogQueryDAO.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.browser.source.BrowserErrorCategory;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLog;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLogs;
+import org.apache.skywalking.oap.server.core.query.type.ErrorCategory;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.library.util.StringUtil;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBBrowserLogQueryDAO implements IBrowserLogQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<BrowserErrorLogRecord> storageBuilder = new BrowserErrorLogRecord.Builder();
+
+ @Override
+ public BrowserErrorLogs queryBrowserErrorLogs(String serviceId, String serviceVersionId, String pagePathId,
+ BrowserErrorCategory category, long startSecondTB,
+ long endSecondTB, int limit, int from) throws IOException {
+ StringBuilder query = new StringBuilder();
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query.append("select * from ");
+ query = client.addModelPath(query, BrowserErrorLogRecord.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ if (StringUtil.isNotEmpty(serviceId)) {
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ }
+ query = client.addQueryIndexValue(BrowserErrorLogRecord.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where 1=1");
+ if (startSecondTB != 0 && endSecondTB != 0) {
+ query.append(" and ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startSecondTB));
+ query.append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endSecondTB));
+ }
+ if (StringUtil.isNotEmpty(serviceVersionId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.SERVICE_VERSION_ID).append(" = \"").append(serviceVersionId).append("\"");
+ }
+ if (StringUtil.isNotEmpty(pagePathId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.PAGE_PATH_ID).append(" = \"").append(pagePathId).append("\"");
+ }
+ if (Objects.nonNull(category)) {
+ query.append(" and ").append(BrowserErrorLogRecord.ERROR_CATEGORY).append(" = ").append(category.getValue());
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString();
+ queryString = queryString.replace("1=1 and ", "");
+
+ List<? super StorageData> storageDataList = client.filterQuery(BrowserErrorLogRecord.INDEX_NAME, queryString, storageBuilder);
+ List<BrowserErrorLogRecord> browserErrorLogRecordList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> browserErrorLogRecordList.add((BrowserErrorLogRecord) storageData));
+ // resort by self, because of the select query result order by time.
+ browserErrorLogRecordList.sort((BrowserErrorLogRecord b1, BrowserErrorLogRecord b2) -> Long.compare(b2.getTimestamp(), b1.getTimestamp()));
+ BrowserErrorLogs logs = new BrowserErrorLogs();
+ int limitCount = 0;
+ for (int i = 0; i < browserErrorLogRecordList.size(); i++) {
+ if (i >= from && limitCount < limit) {
+ limitCount++;
Review comment:
Oh, that's my carelessness. I will fix it tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r757284528
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String GROUP_IDX = "service_group";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getMeasurements(), request.getMeasurementTypes(), request.getMeasurementValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ if (log.isDebugEnabled()) {
+ for (IoTDBInsertRequest request : requestList) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+ }
+
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getMeasurements());
+ typesList.add(request.getMeasurementTypes());
+ valuesList.add(request.getMeasurementValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ SessionDataSetWrapper wrapper = null;
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
Review comment:
Or maybe there are some places where you use `String.valueOf()` without checking nullability
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r757451628
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBInsertRequest.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+import org.apache.skywalking.oap.server.library.client.request.InsertRequest;
+import org.apache.skywalking.oap.server.library.client.request.UpdateRequest;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBIndexes;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+
+@Getter
+@Setter
+@ToString
+@Slf4j
+public class IoTDBInsertRequest implements InsertRequest, UpdateRequest {
+ private String modelName;
+ private long time;
+ private List<String> indexes;
+ private List<String> indexValues;
+ private List<String> measurements;
+ private List<TSDataType> measurementTypes;
+ private List<Object> measurementValues;
+
+ public <T extends StorageData> IoTDBInsertRequest(String modelName, long time, T storageData,
+ StorageHashMapBuilder<T> storageBuilder) {
+ this.modelName = modelName;
+ this.time = time;
+ indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexValues = new ArrayList<>(indexes.size());
+ Map<String, Object> storageMap = storageBuilder.entity2Storage(storageData);
+
+ indexes.forEach(index -> {
+ if (index.equals(IoTDBIndexes.ID_IDX)) {
+ indexValues.add(storageData.id());
+ } else if (storageMap.containsKey(index)) {
+ // avoid `service_group` be "null" when inserting
+ if (index.equals(IoTDBIndexes.GROUP_IDX) && storageMap.get(index) == null) {
Review comment:
If I don't use the null check for `Group_IDX`, `String.valueOf(null)` in line 68 will return "null". In production, all other indexes are xxx_id and they won't be null. But `Group_IDX` maybe null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714959506
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBLogQueryDAO.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.network.logging.v3.LogTags;
+import org.apache.skywalking.apm.util.StringUtil;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.AbstractLogRecord;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.TraceScopeCondition;
+import org.apache.skywalking.oap.server.core.query.type.ContentType;
+import org.apache.skywalking.oap.server.core.query.type.KeyValue;
+import org.apache.skywalking.oap.server.core.query.type.Log;
+import org.apache.skywalking.oap.server.core.query.type.Logs;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.ILogQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBLogQueryDAO implements ILogQueryDAO {
Review comment:
I have cancelled fuzzy query in IoTDBLogQueryDAO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715294986
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,62 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.tomcat.embed</groupId>
+ <artifactId>tomcat-embed-core</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
Review comment:
*Critical OSS Vulnerability:*
### pkg:maven/org.apache.tomcat.embed/tomcat-embed-core
56 Critical, 117 Severe, 13 Moderate, 0 Unknown vulnerabilities have been found across 1 dependencies
<details>
<summary><b>Components</b></summary><br/>
<ul>
<details>
<summary><b>pkg:maven/org.apache.tomcat.embed/tomcat-embed-core</b></summary>
<ul>
<details>
<summary><b>CRITICAL Vulnerabilities (56)</b></summary><br/>
<ul>
<details>
<summary>CVE-2016-8735</summary>
> #### [CVE-2016-8735] Improper Access Control
> Remote code execution is possible with Apache Tomcat before 6.0.48, 7.x before 7.0.73, 8.x before 8.0.39, 8.5.x before 8.5.7, and 9.x before 9.0.0.M12 if JmxRemoteLifecycleListener is used and an attacker can reach JMX ports. The issue exists because this listener wasn't updated for consistency with the CVE-2016-3427 Oracle patch that affected credential types.
>
> **CVSS Score:** 9.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2017-5651</summary>
> #### [CVE-2017-5651] Data Handling
> In Apache Tomcat 9.0.0.M1 to 9.0.0.M18 and 8.5.0 to 8.5.12, the refactoring of the HTTP connectors introduced a regression in the send file processing. If the send file processing completed quickly, it was possible for the Processor to be added to the processor cache twice. This could result in the same Processor being used for multiple requests which in turn could lead to unexpected errors and/or response mix-up.
>
> **CVSS Score:** 9.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2018-8014</summary>
> #### [CVE-2018-8014] The defaults settings for the CORS filter provided in Apache Tomcat 9.0.0.M1 to ...
> The defaults settings for the CORS filter provided in Apache Tomcat 9.0.0.M1 to 9.0.8, 8.5.0 to 8.5.31, 8.0.0.RC1 to 8.0.52, 7.0.41 to 7.0.88 are insecure and enable 'supportsCredentials' for all origins. It is expected that users of the CORS filter will have configured it appropriately for their environment rather than using it in the default configuration. Therefore, it is expected that most users will not be impacted by this issue.
>
> **CVSS Score:** 9.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2020-1938</summary>
> #### [CVE-2020-1938] When using the Apache JServ Protocol (AJP), care must be taken when trusting inc...
> When using the Apache JServ Protocol (AJP), care must be taken when trusting incoming connections to Apache Tomcat. Tomcat treats AJP connections as having higher trust than, for example, a similar HTTP connection. If such connections are available to an attacker, they can be exploited in ways that may be surprising. In Apache Tomcat 9.0.0.M1 to 9.0.0.30, 8.5.0 to 8.5.50 and 7.0.0 to 7.0.99, Tomcat shipped with an AJP Connector enabled by default that listened on all configured IP addresses. It was expected (and recommended in the security guide) that this Connector would be disabled if not required. This vulnerability report identified a mechanism that allowed: - returning arbitrary files from anywhere in the web application - processing any file in the web application as a JSP Further, if the web application allowed file upload and stored those files within the web application (or the attacker was able to control the content of the web application by some other means) then thi
s, along with the ability to process a file as a JSP, made remote code execution possible. It is important to note that mitigation is only required if an AJP port is accessible to untrusted users. Users wishing to take a defence-in-depth approach and block the vector that permits returning arbitrary files and execution as JSP may upgrade to Apache Tomcat 9.0.31, 8.5.51 or 7.0.100 or later. A number of changes were made to the default AJP Connector configuration in 9.0.31 to harden the default configuration. It is likely that users upgrading to 9.0.31, 8.5.51 or 7.0.100 or later will need to make small changes to their configurations.
>
> **CVSS Score:** 9.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2017-5648</summary>
> #### [CVE-2017-5648] Improper Access Control
> While investigating bug 60718, it was noticed that some calls to application listeners in Apache Tomcat 9.0.0.M1 to 9.0.0.M17, 8.5.0 to 8.5.11, 8.0.0.RC1 to 8.0.41, and 7.0.0 to 7.0.75 did not use the appropriate facade object. When running an untrusted application under a SecurityManager, it was therefore possible for that untrusted application to retain a reference to the request or response object and thereby access and/or modify information associated with another web application.
>
> **CVSS Score:** 9.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N
</details>
<details>
<summary>CVE-2016-0714</summary>
> #### [CVE-2016-0714] Permissions, Privileges, and Access Controls
> The session-persistence implementation in Apache Tomcat 6.x before 6.0.45, 7.x before 7.0.68, 8.x before 8.0.31, and 9.x before 9.0.0.M2 mishandles session attributes, which allows remote authenticated users to bypass intended SecurityManager restrictions and execute arbitrary code in a privileged context via a web application that places a crafted object in a session.
>
> **CVSS Score:** 8.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2015-5351</summary>
> #### [CVE-2015-5351] Cross-Site Request Forgery (CSRF)
> The (1) Manager and (2) Host Manager applications in Apache Tomcat 7.x before 7.0.68, 8.x before 8.0.31, and 9.x before 9.0.0.M2 establish sessions and send CSRF tokens for arbitrary new requests, which allows remote attackers to bypass a CSRF protection mechanism by using a token.
>
> **CVSS Score:** 8.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2021-30640</summary>
> #### [CVE-2021-30640] A vulnerability in the JNDI Realm of Apache Tomcat allows an attacker to authent...
> A vulnerability in the JNDI Realm of Apache Tomcat allows an attacker to authenticate using variations of a valid user name and/or to bypass some of the protection provided by the LockOut Realm. This issue affects Apache Tomcat 10.0.0-M1 to 10.0.5; 9.0.0.M1 to 9.0.45; 8.5.0 to 8.5.65.
>
> **CVSS Score:** 8.2
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:H/A:N
</details>
<details>
<summary>CVE-2019-0232</summary>
> #### [CVE-2019-0232] Improper Input Validation
> When running on Windows with enableCmdLineArguments enabled, the CGI Servlet in Apache Tomcat 9.0.0.M1 to 9.0.17, 8.5.0 to 8.5.39 and 7.0.0 to 7.0.93 is vulnerable to Remote Code Execution due to a bug in the way the JRE passes command line arguments to Windows. The CGI Servlet is disabled by default. The CGI option enableCmdLineArguments is disable by default in Tomcat 9.0.x (and will be disabled by default in all versions in response to this vulnerability). For a detailed explanation of the JRE behaviour, see Markus Wulftange's blog (https://codewhitesec.blogspot.com/2016/02/java-and-command-line-injections-in-windows.html) and this archived MSDN blog (https://web.archive.org/web/20161228144344/https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/everyone-quotes-command-line-arguments-the-wrong-way/).
>
> **CVSS Score:** 8.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2017-12615</summary>
> #### [CVE-2017-12615] When running Apache Tomcat 7.0.0 to 7.0.79 on Windows with HTTP PUTs enabled (e....
> When running Apache Tomcat 7.0.0 to 7.0.79 on Windows with HTTP PUTs enabled (e.g. via setting the readonly initialisation parameter of the Default to false) it was possible to upload a JSP file to the server via a specially crafted request. This JSP could then be requested and any code it contained would be executed by the server.
>
> **CVSS Score:** 8.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2017-12617</summary>
> #### [CVE-2017-12617] When running Apache Tomcat versions 9.0.0.M1 to 9.0.0, 8.5.0 to 8.5.22, 8.0.0.RC...
> When running Apache Tomcat versions 9.0.0.M1 to 9.0.0, 8.5.0 to 8.5.22, 8.0.0.RC1 to 8.0.46 and 7.0.0 to 7.0.81 with HTTP PUTs enabled (e.g. via setting the readonly initialisation parameter of the Default servlet to false) it was possible to upload a JSP file to the server via a specially crafted request. This JSP could then be requested and any code it contained would be executed by the server.
>
> **CVSS Score:** 8.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2015-5346</summary>
> #### [CVE-2015-5346] Session fixation vulnerability in Apache Tomcat 7.x before 7.0.66, 8.x before 8....
> Session fixation vulnerability in Apache Tomcat 7.x before 7.0.66, 8.x before 8.0.30, and 9.x before 9.0.0.M2, when different session settings are used for deployments of multiple versions of the same web application, might allow remote attackers to hijack web sessions by leveraging use of a requestedSessionSSL field for an unintended request, related to CoyoteAdapter.java and Request.java.
>
> **CVSS Score:** 8.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2016-5388</summary>
> #### [CVE-2016-5388] Improper Access Control
> Apache Tomcat through 8.5.4, when the CGI Servlet is enabled, follows RFC 3875 section 4.1.18 and therefore does not protect applications from the presence of untrusted client data in the HTTP_PROXY environment variable, which might allow remote attackers to redirect an application's outbound HTTP traffic to an arbitrary proxy server via a crafted Proxy header in an HTTP request, aka an "httpoxy" issue. NOTE: the vendor states "A mitigation is planned for future releases of Tomcat, tracked as CVE-2016-5388"; in other words, this is not a CVE ID for a vulnerability.
>
> **CVSS Score:** 8.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2002-2272</summary>
> #### [CVE-2002-2272] Improper Restriction of Operations within the Bounds of a Memory Buffer
> Tomcat 4.0 through 4.1.12, using mod_jk 1.2.1 module on Apache 1.3 through 1.3.27, allows remote attackers to cause a denial of service (desynchronized communications) via an HTTP GET request with a Transfer-Encoding chunked field with invalid values.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:C
</details>
<details>
<summary>CVE-2005-4836</summary>
> #### [CVE-2005-4836] Information Exposure
> The HTTP/1.1 connector in Apache Tomcat 4.1.15 through 4.1.40 does not reject NULL bytes in a URL when allowLinking is configured, which allows remote attackers to read JSP source files and obtain sensitive information.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:C/I:N/A:N
</details>
<details>
<summary>CVE-2006-7197</summary>
> #### [CVE-2006-7197] The AJP connector in Apache Tomcat 5.5.15 uses an incorrect length for chunks, w...
> The AJP connector in Apache Tomcat 5.5.15 uses an incorrect length for chunks, which can cause a buffer over-read in the ajp_process_callback in mod_jk, which allows remote attackers to read portions of sensitive memory.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:C/I:N/A:N
</details>
<details>
<summary>CVE-2016-9774</summary>
> #### [CVE-2016-9774] Improper Link Resolution Before File Access ("Link Following")
> The postinst script in the tomcat6 package before 6.0.45+dfsg-1~deb7u4 on Debian wheezy, before 6.0.35-1ubuntu3.9 on Ubuntu 12.04 LTS and on Ubuntu 14.04 LTS; the tomcat7 package before 7.0.28-4+deb7u8 on Debian wheezy, before 7.0.56-3+deb8u6 on Debian jessie, before 7.0.52-1ubuntu0.8 on Ubuntu 14.04 LTS, and on Ubuntu 12.04 LTS, 16.04 LTS, and 16.10; and the tomcat8 package before 8.0.14-1+deb8u5 on Debian jessie, before 8.0.32-1ubuntu1.3 on Ubuntu 16.04 LTS, before 8.0.37-1ubuntu0.1 on Ubuntu 16.10, and before 8.0.38-2ubuntu1 on Ubuntu 17.04 might allow local users with access to the tomcat account to obtain sensitive information or gain root privileges via a symlink attack on the Catalina localhost directory.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2014-0230</summary>
> #### [CVE-2014-0230] Resource Management Errors
> Apache Tomcat 6.x before 6.0.44, 7.x before 7.0.55, and 8.x before 8.0.9 does not properly handle cases where an HTTP response occurs before finishing the reading of an entire request body, which allows remote attackers to cause a denial of service (thread consumption) via a series of aborted upload attempts.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:C
</details>
<details>
<summary>CVE-2016-1240</summary>
> #### [CVE-2016-1240] Improper Input Validation
> The Tomcat init script in the tomcat7 package before 7.0.56-3+deb8u4 and tomcat8 package before 8.0.14-1+deb8u3 on Debian jessie and the tomcat6 and libtomcat6-java packages before 6.0.35-1ubuntu3.8 on Ubuntu 12.04 LTS, the tomcat7 and libtomcat7-java packages before 7.0.52-1ubuntu0.7 on Ubuntu 14.04 LTS, and tomcat8 and libtomcat8-java packages before 8.0.32-1ubuntu1.2 on Ubuntu 16.04 LTS allows local users with access to the tomcat account to gain root privileges via a symlink attack on the Catalina log file, as demonstrated by /var/log/tomcat7/catalina.out.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2016-9775</summary>
> #### [CVE-2016-9775] Permissions, Privileges, and Access Controls
> The postrm script in the tomcat6 package before 6.0.45+dfsg-1~deb7u3 on Debian wheezy, before 6.0.45+dfsg-1~deb8u1 on Debian jessie, before 6.0.35-1ubuntu3.9 on Ubuntu 12.04 LTS and on Ubuntu 14.04 LTS; the tomcat7 package before 7.0.28-4+deb7u7 on Debian wheezy, before 7.0.56-3+deb8u6 on Debian jessie, before 7.0.52-1ubuntu0.8 on Ubuntu 14.04 LTS, and on Ubuntu 12.04 LTS, 16.04 LTS, and 16.10; and the tomcat8 package before 8.0.14-1+deb8u5 on Debian jessie, before 8.0.32-1ubuntu1.3 on Ubuntu 16.04 LTS, before 8.0.37-1ubuntu0.1 on Ubuntu 16.10, and before 8.0.38-2ubuntu1 on Ubuntu 17.04 might allow local users with access to the tomcat account to gain root privileges via a setgid program in the Catalina directory, as demonstrated by /etc/tomcat8/Catalina/attack.
>
> **CVSS Score:** 7.8
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2002-0493</summary>
> #### [CVE-2002-0493] null
> Apache Tomcat may be started without proper security settings if errors are encountered while reading the web.xml file, which could allow attackers to bypass intended restrictions.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2018-8034</summary>
> #### [CVE-2018-8034] The host name verification when using TLS with the WebSocket client was missing....
> The host name verification when using TLS with the WebSocket client was missing. It is now enabled by default. Versions Affected: Apache Tomcat 9.0.0.M1 to 9.0.9, 8.5.0 to 8.5.31, 8.0.0.RC1 to 8.0.52, and 7.0.35 to 7.0.88.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2017-5664</summary>
> #### [CVE-2017-5664] The error page mechanism of the Java Servlet Specification requires that, when a...
> The error page mechanism of the Java Servlet Specification requires that, when an error occurs and an error page is configured for the error that occurred, the original request and response are forwarded to the error page. This means that the request is presented to the error page with the original HTTP method. If the error page is a static file, expected behaviour is to serve content of the file as if processing a GET request, regardless of the actual HTTP method. The Default Servlet in Apache Tomcat 9.0.0.M1 to 9.0.0.M20, 8.5.0 to 8.5.14, 8.0.0.RC1 to 8.0.43 and 7.0.0 to 7.0.77 did not do this. Depending on the original request this could lead to unexpected and undesirable results for static error pages including, if the DefaultServlet is configured to permit writes, the replacement or removal of the custom error page. Notes for other user provided error pages: (1) Unless explicitly coded otherwise, JSPs ignore the HTTP method. JSPs used as error pages must must ensure that t
hey handle any error dispatch as a GET request, regardless of the actual method. (2) By default, the response generated by a Servlet does depend on the HTTP method. Custom Servlets used as error pages must ensure that they handle any error dispatch as a GET request, regardless of the actual method.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N
</details>
<details>
<summary>CVE-2016-6797</summary>
> #### [CVE-2016-6797] Improper Access Control
> The ResourceLinkFactory implementation in Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0.0 to 7.0.70 and 6.0.0 to 6.0.45 did not limit web application access to global JNDI resources to those resources explicitly linked to the web application. Therefore, it was possible for a web application to access any global JNDI resource whether an explicit ResourceLink had been configured or not.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2002-1394</summary>
> #### [CVE-2002-1394] Apache Tomcat 4.0.5 and earlier, when using both the invoker servlet and the def...
> Apache Tomcat 4.0.5 and earlier, when using both the invoker servlet and the default servlet, allows remote attackers to read source code for server files or bypass certain protections, a variant of CAN-2002-1148.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2016-3092</summary>
> #### [CVE-2016-3092] Improper Input Validation
> The MultipartStream class in Apache Commons Fileupload before 1.3.2, as used in Apache Tomcat 7.x before 7.0.70, 8.x before 8.0.36, 8.5.x before 8.5.3, and 9.x before 9.0.0.M7 and other products, allows remote attackers to cause a denial of service (CPU consumption) via a long boundary string.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2016-6817</summary>
> #### [CVE-2016-6817] Improper Restriction of Operations within the Bounds of a Memory Buffer
> The HTTP/2 header parser in Apache Tomcat 9.0.0.M1 to 9.0.0.M11 and 8.5.0 to 8.5.6 entered an infinite loop if a header was received that was larger than the available buffer. This made a denial of service attack possible.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2014-0050</summary>
> #### [CVE-2014-0050] Permissions, Privileges, and Access Controls
> MultipartStream.java in Apache Commons FileUpload before 1.3.1, as used in Apache Tomcat, JBoss Web, and other products, allows remote attackers to cause a denial of service (infinite loop and CPU consumption) via a crafted Content-Type header that bypasses a loop's intended exit conditions.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2017-12616</summary>
> #### [CVE-2017-12616] When using a VirtualDirContext with Apache Tomcat 7.0.0 to 7.0.80 it was possibl...
> When using a VirtualDirContext with Apache Tomcat 7.0.0 to 7.0.80 it was possible to bypass security constraints and/or view the source code of JSPs for resources served by the VirtualDirContext using a specially crafted request.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2020-13935</summary>
> #### [CVE-2020-13935] The payload length in a WebSocket frame was not correctly validated in Apache To...
> The payload length in a WebSocket frame was not correctly validated in Apache Tomcat 10.0.0-M1 to 10.0.0-M6, 9.0.0.M1 to 9.0.36, 8.5.0 to 8.5.56 and 7.0.27 to 7.0.104. Invalid payload lengths could trigger an infinite loop. Multiple requests with invalid payload lengths could lead to a denial of service.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2019-17563</summary>
> #### [CVE-2019-17563] When using FORM authentication with Apache Tomcat 9.0.0.M1 to 9.0.29, 8.5.0 to 8...
> When using FORM authentication with Apache Tomcat 9.0.0.M1 to 9.0.29, 8.5.0 to 8.5.49 and 7.0.0 to 7.0.98 there was a narrow window where an attacker could perform a session fixation attack. The window was considered too narrow for an exploit to be practical but, erring on the side of caution, this issue has been treated as a security vulnerability.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:R/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2016-6796</summary>
> #### [CVE-2016-6796] A malicious web application running on Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0...
> A malicious web application running on Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0.0 to 7.0.70 and 6.0.0 to 6.0.45 was able to bypass a configured SecurityManager via manipulation of the configuration parameters for the JSP Servlet.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N
</details>
<details>
<summary>CVE-2017-5650</summary>
> #### [CVE-2017-5650] Resource Management Errors
> In Apache Tomcat 9.0.0.M1 to 9.0.0.M18 and 8.5.0 to 8.5.12, the handling of an HTTP/2 GOAWAY frame for a connection did not close streams associated with that connection that were currently waiting for a WINDOW_UPDATE before allowing the application to write more data. These waiting streams each consumed a thread. A malicious client could therefore construct a series of HTTP/2 requests that would consume all available processing threads.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2020-13934</summary>
> #### [CVE-2020-13934] An h2c direct connection to Apache Tomcat 10.0.0-M1 to 10.0.0-M6, 9.0.0.M5 to 9....
> An h2c direct connection to Apache Tomcat 10.0.0-M1 to 10.0.0-M6, 9.0.0.M5 to 9.0.36 and 8.5.1 to 8.5.56 did not release the HTTP/1.1 processor after the upgrade to HTTP/2. If a sufficient number of such requests were made, an OutOfMemoryException could occur leading to a denial of service.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2017-7675</summary>
> #### [CVE-2017-7675] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> The HTTP/2 implementation in Apache Tomcat 9.0.0.M1 to 9.0.0.M21 and 8.5.0 to 8.5.15 bypassed a number of security checks that prevented directory traversal attacks. It was therefore possible to bypass security constraints using a specially crafted URL.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2020-17527</summary>
> #### [CVE-2020-17527] While investigating bug 64830 it was discovered that Apache Tomcat 10.0.0-M1 to ...
> While investigating bug 64830 it was discovered that Apache Tomcat 10.0.0-M1 to 10.0.0-M9, 9.0.0-M1 to 9.0.39 and 8.5.0 to 8.5.59 could re-use an HTTP request header value from the previous stream received on an HTTP/2 connection for the request associated with the subsequent stream. While this would most likely lead to an error and the closure of the HTTP/2 connection, it is possible that information could leak between requests.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2018-1336</summary>
> #### [CVE-2018-1336] Uncontrolled Resource Consumption ("Resource Exhaustion")
> An improper handing of overflow in the UTF-8 decoder with supplementary characters can lead to an infinite loop in the decoder causing a Denial of Service. Versions Affected: Apache Tomcat 9.0.0.M9 to 9.0.7, 8.5.0 to 8.5.30, 8.0.0.RC1 to 8.0.51, and 7.0.28 to 7.0.86.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2013-2185</summary>
> #### [CVE-2013-2185] Improper Input Validation
> ** DISPUTED ** The readObject method in the DiskFileItem class in Apache Tomcat and JBoss Web, as used in Red Hat JBoss Enterprise Application Platform 6.1.0 and Red Hat JBoss Portal 6.0.0, allows remote attackers to write to arbitrary files via a NULL byte in a file name in a serialized instance, a similar issue to CVE-2013-2186. NOTE: this issue is reportedly disputed by the Apache Tomcat team, although Red Hat considers it a vulnerability. The dispute appears to regard whether it is the responsibility of applications to avoid providing untrusted data to be deserialized, or whether this class should inherently protect against this issue.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2020-11996</summary>
> #### [CVE-2020-11996] A specially crafted sequence of HTTP/2 requests sent to Apache Tomcat 10.0.0-M1 ...
> A specially crafted sequence of HTTP/2 requests sent to Apache Tomcat 10.0.0-M1 to 10.0.0-M5, 9.0.0.M1 to 9.0.35 and 8.5.0 to 8.5.55 could trigger high CPU usage for several seconds. If a sufficient number of such requests were made on concurrent HTTP/2 connections, the server could become unresponsive.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2021-25122</summary>
> #### [CVE-2021-25122] When responding to new h2c connection requests, Apache Tomcat versions 10.0.0-M1...
> When responding to new h2c connection requests, Apache Tomcat versions 10.0.0-M1 to 10.0.0, 9.0.0.M1 to 9.0.41 and 8.5.0 to 8.5.61 could duplicate request headers and a limited amount of request body from one request to another meaning user A and user B could both see the results of user A's request.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2001-1563</summary>
> #### [CVE-2001-1563] Unknown vulnerability in Tomcat 3.2.1 running on HP Secure OS for Linux 1.0 allo...
> Unknown vulnerability in Tomcat 3.2.1 running on HP Secure OS for Linux 1.0 allows attackers to access servlet resources. NOTE: due to the vagueness of the vendor advisory, it is not clear whether this issue is already covered by other CVE identifiers.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2009-3548</summary>
> #### [CVE-2009-3548] Credentials Management
> The Windows installer for Apache Tomcat 6.0.0 through 6.0.20, 5.5.0 through 5.5.28, and possibly earlier versions uses a blank default password for the administrative user, which allows remote attackers to gain privileges.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2021-24122</summary>
> #### [CVE-2021-24122] When serving resources from a network location using the NTFS file system, Apach...
> When serving resources from a network location using the NTFS file system, Apache Tomcat versions 10.0.0-M1 to 10.0.0-M9, 9.0.0.M1 to 9.0.39, 8.5.0 to 8.5.59 and 7.0.0 to 7.0.106 were susceptible to JSP source code disclosure in some configurations. The root cause was the unexpected behaviour of the JRE API File.getCanonicalPath() which in turn was caused by the inconsistent behaviour of the Windows API (FindFirstFileW) in some circumstances.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2002-0682</summary>
> #### [CVE-2002-0682] Cross-site scripting vulnerability in Apache Tomcat 4.0.3 allows remote attacker...
> Cross-site scripting vulnerability in Apache Tomcat 4.0.3 allows remote attackers to execute script as other web users via script in a URL with the /servlet/ mapping, which does not filter the script when an exception is thrown by the servlet.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2021-30639</summary>
> #### [CVE-2021-30639] A vulnerability in Apache Tomcat allows an attacker to remotely trigger a denial...
> A vulnerability in Apache Tomcat allows an attacker to remotely trigger a denial of service. An error introduced as part of a change to improve error handling during non-blocking I/O meant that the error flag associated with the Request object was not reset between requests. This meant that once a non-blocking I/O error occurred, all future requests handled by that request object would fail. Users were able to trigger non-blocking I/O errors, e.g. by dropping a connection, thereby creating the possibility of triggering a DoS. Applications that do not use non-blocking I/O are not exposed to this vulnerability. This issue affects Apache Tomcat 10.0.3 to 10.0.4; 9.0.44; 8.5.64.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2016-5018</summary>
> #### [CVE-2016-5018] In Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0....
> In Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0.0 to 7.0.70 and 6.0.0 to 6.0.45 a malicious web application was able to bypass a configured SecurityManager via a Tomcat utility method that was accessible to web applications.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N
</details>
<details>
<summary>CVE-2011-3190</summary>
> #### [CVE-2011-3190] Permissions, Privileges, and Access Controls
> Certain AJP protocol connector implementations in Apache Tomcat 7.0.0 through 7.0.20, 6.0.0 through 6.0.33, 5.5.0 through 5.5.33, and possibly other versions allow remote attackers to spoof AJP requests, bypass authentication, and obtain sensitive information by causing the connector to interpret a request body as a new request.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2019-0199</summary>
> #### [CVE-2019-0199] Uncontrolled Resource Consumption ("Resource Exhaustion")
> The HTTP/2 implementation in Apache Tomcat 9.0.0.M1 to 9.0.14 and 8.5.0 to 8.5.37 accepted streams with excessive numbers of SETTINGS frames and also permitted clients to keep streams open without reading/writing request/response data. By keeping streams open for requests that utilised the Servlet API's blocking I/O, clients were able to cause server-side threads to block eventually leading to thread exhaustion and a DoS.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2017-5647</summary>
> #### [CVE-2017-5647] Information Exposure
> A bug in the handling of the pipelined requests in Apache Tomcat 9.0.0.M1 to 9.0.0.M18, 8.5.0 to 8.5.12, 8.0.0.RC1 to 8.0.42, 7.0.0 to 7.0.76, and 6.0.0 to 6.0.52, when send file was used, results in the pipelined request being lost when send file processing of the previous request completed. This could result in responses appearing to be sent for the wrong request. For example, a user agent that sent requests A, B and C could see the correct response for request A, the response for request C for request B and no response for request C.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2016-8745</summary>
> #### [CVE-2016-8745] A bug in the error handling of the send file code for the NIO HTTP connector in ...
> A bug in the error handling of the send file code for the NIO HTTP connector in Apache Tomcat 9.0.0.M1 to 9.0.0.M13, 8.5.0 to 8.5.8, 8.0.0.RC1 to 8.0.39, 7.0.0 to 7.0.73 and 6.0.16 to 6.0.48 resulted in the current Processor object being added to the Processor cache multiple times. This in turn meant that the same Processor could be used for concurrent requests. Sharing a Processor can result in information leakage between requests including, not not limited to, session ID and the response body. The bug was first noticed in 8.5.x onwards where it appears the refactoring of the Connector code for 8.5.x onwards made it more likely that the bug was observed. Initially it was thought that the 8.5.x refactoring introduced the bug but further investigation has shown that the bug is present in all currently supported Tomcat versions.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2019-10072</summary>
> #### [CVE-2019-10072] Uncontrolled Resource Consumption ("Resource Exhaustion")
> The fix for CVE-2019-0199 was incomplete and did not address HTTP/2 connection window exhaustion on write in Apache Tomcat versions 9.0.0.M1 to 9.0.19 and 8.5.0 to 8.5.40 . By not sending WINDOW_UPDATE messages for the connection window (stream 0) clients were able to cause server-side threads to block eventually leading to thread exhaustion and a DoS.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2016-8747</summary>
> #### [CVE-2016-8747] Information Exposure
> An information disclosure issue was discovered in Apache Tomcat 8.5.7 to 8.5.9 and 9.0.0.M11 to 9.0.0.M15 in reverse-proxy configurations. Http11InputBuffer.java allows remote attackers to read data that was intended to be associated with a different request.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2016-6816</summary>
> #### [CVE-2016-6816] Improper Input Validation
> The code in Apache Tomcat 9.0.0.M1 to 9.0.0.M11, 8.5.0 to 8.5.6, 8.0.0.RC1 to 8.0.38, 7.0.0 to 7.0.72, and 6.0.0 to 6.0.47 that parsed the HTTP request line permitted invalid characters. This could be exploited, in conjunction with a proxy that also permitted the invalid characters but with a different interpretation, to inject data into the HTTP response. By manipulating the HTTP response the attacker could poison a web-cache, perform an XSS attack and/or obtain sensitive information from requests other then their own.
>
> **CVSS Score:** 7.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:C/C:L/I:L/A:L
</details>
<details>
<summary>CVE-2020-9484</summary>
> #### [CVE-2020-9484] When using Apache Tomcat versions 10.0.0-M1 to 10.0.0-M4, 9.0.0.M1 to 9.0.34, 8....
> When using Apache Tomcat versions 10.0.0-M1 to 10.0.0-M4, 9.0.0.M1 to 9.0.34, 8.5.0 to 8.5.54 and 7.0.0 to 7.0.103 if a) an attacker is able to control the contents and name of a file on the server; and b) the server is configured to use the PersistenceManager with a FileStore; and c) the PersistenceManager is configured with sessionAttributeValueClassNameFilter="null" (the default unless a SecurityManager is used) or a sufficiently lax filter to allow the attacker provided object to be deserialized; and d) the attacker knows the relative file path from the storage location used by FileStore to the file the attacker has control over; then, using a specifically crafted request, the attacker will be able to trigger remote code execution via deserialization of the file under their control. Note that all of conditions a) to d) must be true for the attack to succeed.
>
> **CVSS Score:** 7
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2021-25329</summary>
> #### [CVE-2021-25329] The fix for CVE-2020-9484 was incomplete. When using Apache Tomcat 10.0.0-M1 to ...
> The fix for CVE-2020-9484 was incomplete. When using Apache Tomcat 10.0.0-M1 to 10.0.0, 9.0.0.M1 to 9.0.41, 8.5.0 to 8.5.61 or 7.0.0. to 7.0.107 with a configuration edge case that was highly unlikely to be used, the Tomcat instance was still vulnerable to CVE-2020-9494. Note that both the previously published prerequisites for CVE-2020-9484 and the previously published mitigations for CVE-2020-9484 also apply to this issue.
>
> **CVSS Score:** 7
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2019-12418</summary>
> #### [CVE-2019-12418] When Apache Tomcat 9.0.0.M1 to 9.0.28, 8.5.0 to 8.5.47, 7.0.0 and 7.0.97 is conf...
> When Apache Tomcat 9.0.0.M1 to 9.0.28, 8.5.0 to 8.5.47, 7.0.0 and 7.0.97 is configured with the JMX Remote Lifecycle Listener, a local attacker without access to the Tomcat process or configuration files is able to manipulate the RMI registry to perform a man-in-the-middle attack to capture user names and passwords used to access the JMX interface. The attacker can then use these credentials to access the JMX interface and gain complete control over the Tomcat instance.
>
> **CVSS Score:** 7
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
</ul>
</details>
<details>
<summary><b>SEVERE Vulnerabilities (117)</b></summary><br/>
<ul>
<details>
<summary>CVE-2013-6357</summary>
> #### [CVE-2013-6357] Cross-Site Request Forgery (CSRF)
> ** DISPUTED ** Cross-site request forgery (CSRF) vulnerability in the Manager application in Apache Tomcat 5.5.25 and earlier allows remote attackers to hijack the authentication of administrators for requests that manipulate application deployment via the POST method, as demonstrated by a /manager/html/undeploy?path= URI. NOTE: the vendor disputes the significance of this report, stating that "the Apache Tomcat Security team has not accepted any reports of CSRF attacks against the Manager application ... as they require a reckless system administrator."
>
> **CVSS Score:** 6.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2002-1567</summary>
> #### [CVE-2002-1567] Cross-site scripting (XSS) vulnerability in Apache Tomcat 4.1 allows remote atta...
> Cross-site scripting (XSS) vulnerability in Apache Tomcat 4.1 allows remote attackers to execute arbitrary web script and steal cookies via a URL with encoded newlines followed by a request to a .jsp file whose name contains the script.
>
> **CVSS Score:** 6.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2013-2067</summary>
> #### [CVE-2013-2067] Improper Authentication
> java/org/apache/catalina/authenticator/FormAuthenticator.java in the form authentication feature in Apache Tomcat 6.0.21 through 6.0.36 and 7.x before 7.0.33 does not properly handle the relationships between authentication requirements and sessions, which allows remote attackers to inject a request into a session by sending this request during completion of the login form, a variant of a session fixation attack.
>
> **CVSS Score:** 6.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2013-4444</summary>
> #### [CVE-2013-4444] Improper Control of Generation of Code ("Code Injection")
> Unrestricted file upload vulnerability in Apache Tomcat 7.x before 7.0.40, in certain situations involving outdated java.io.File code and a custom JMX configuration, allows remote attackers to execute arbitrary code by uploading and accessing a JSP file.
>
> **CVSS Score:** 6.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2003-0044</summary>
> #### [CVE-2003-0044] Multiple cross-site scripting (XSS) vulnerabilities in the (1) examples and (2) ...
> Multiple cross-site scripting (XSS) vulnerabilities in the (1) examples and (2) ROOT web applications for Jakarta Tomcat 3.x through 3.3.1a allow remote attackers to insert arbitrary web script or HTML.
>
> **CVSS Score:** 6.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2018-1305</summary>
> #### [CVE-2018-1305] Improper Access Control
> Security constraints defined by annotations of Servlets in Apache Tomcat 9.0.0.M1 to 9.0.4, 8.5.0 to 8.5.27, 8.0.0.RC1 to 8.0.49 and 7.0.0 to 7.0.84 were only applied once a Servlet had been loaded. Because security constraints defined in this way apply to the URL pattern and any URLs below that point, it was possible - depending on the order Servlets were loaded - for some security constraints not to be applied. This could have exposed resources to users who were not authorised to access them.
>
> **CVSS Score:** 6.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2010-2227</summary>
> #### [CVE-2010-2227] Improper Restriction of Operations within the Bounds of a Memory Buffer
> Apache Tomcat 5.5.0 through 5.5.29, 6.0.0 through 6.0.27, and 7.0.0 beta does not properly handle an invalid Transfer-Encoding header, which allows remote attackers to cause a denial of service (application outage) or obtain sensitive information via a crafted header that interferes with "recycling of a buffer."
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:P
</details>
<details>
<summary>CVE-2007-5342</summary>
> #### [CVE-2007-5342] Permissions, Privileges, and Access Controls
> The default catalina.policy in the JULI logging component in Apache Tomcat 5.5.9 through 5.5.25 and 6.0.0 through 6.0.15 does not restrict certain permissions for web applications, which allows attackers to modify logging configuration options and overwrite arbitrary files, as demonstrated by changing the (1) level, (2) directory, and (3) prefix attributes in the org.apache.juli.FileHandler handler.
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2000-0759</summary>
> #### [CVE-2000-0759] Jakarta Tomcat 3.1 under Apache reveals physical path information when a remote ...
> Jakarta Tomcat 3.1 under Apache reveals physical path information when a remote attacker requests a URL that does not exist, which generates an error message that includes the physical path.
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2010-4312</summary>
> #### [CVE-2010-4312] null
> The default configuration of Apache Tomcat 6.x does not include the HTTPOnly flag in a Set-Cookie header, which makes it easier for remote attackers to hijack a session via script access to a cookie.
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:P/A:P
</details>
<details>
<summary>CVE-2000-0760</summary>
> #### [CVE-2000-0760] The Snoop servlet in Jakarta Tomcat 3.1 and 3.0 under Apache reveals sensitive s...
> The Snoop servlet in Jakarta Tomcat 3.1 and 3.0 under Apache reveals sensitive system information when a remote attacker requests a nonexistent URL with a .snp extension.
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2014-0227</summary>
> #### [CVE-2014-0227] Data Handling
> java/org/apache/coyote/http11/filters/ChunkedInputFilter.java in Apache Tomcat 6.x before 6.0.42, 7.x before 7.0.55, and 8.x before 8.0.9 does not properly handle attempts to continue reading data after an error has occurred, which allows remote attackers to conduct HTTP request smuggling attacks or cause a denial of service (resource consumption) by streaming data with malformed chunked transfer coding.
>
> **CVSS Score:** 6.4
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:P/A:P
</details>
<details>
<summary>CVE-2016-0763</summary>
> #### [CVE-2016-0763] Permissions, Privileges, and Access Controls
> The setGlobalContext method in org/apache/naming/factory/ResourceLinkFactory.java in Apache Tomcat 7.x before 7.0.68, 8.x before 8.0.31, and 9.x before 9.0.0.M3 does not consider whether ResourceLinkFactory.setGlobalContext callers are authorized, which allows remote authenticated users to bypass intended SecurityManager restrictions and read or write to arbitrary application data, or cause a denial of service (application disruption), via a web application that sets a crafted global context.
>
> **CVSS Score:** 6.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:L
</details>
<details>
<summary>CVE-2019-0221</summary>
> #### [CVE-2019-0221] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> The SSI printenv command in Apache Tomcat 9.0.0.M1 to 9.0.0.17, 8.5.0 to 8.5.39 and 7.0.0 to 7.0.93 echoes user provided data without escaping and is, therefore, vulnerable to XSS. SSI is disabled by default. The printenv command is intended for debugging and is unlikely to be present in a production website.
>
> **CVSS Score:** 6.1
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:C/C:L/I:L/A:N
</details>
<details>
<summary>CVE-2018-1304</summary>
> #### [CVE-2018-1304] The URL pattern of "" (the empty string) which exactly maps to the context root ...
> The URL pattern of "" (the empty string) which exactly maps to the context root was not correctly handled in Apache Tomcat 9.0.0.M1 to 9.0.4, 8.5.0 to 8.5.27, 8.0.0.RC1 to 8.0.49 and 7.0.0 to 7.0.84 when used as part of a security constraint definition. This caused the constraint to be ignored. It was, therefore, possible for unauthorised users to gain access to web application resources that should have been protected. Only security constraints with a URL pattern of the empty string were affected.
>
> **CVSS Score:** 5.9
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2018-8037</summary>
> #### [CVE-2018-8037] Concurrent Execution using Shared Resource with Improper Synchronization ("Race Condition")
> If an async request was completed by the application at the same time as the container triggered the async timeout, a race condition existed that could result in a user seeing a response intended for a different user. An additional issue was present in the NIO and NIO2 connectors that did not correctly track the closure of the connection when an async request was completed by the application and timed out by the container at the same time. This could also result in a user seeing a response intended for another user. Versions Affected: Apache Tomcat 9.0.0.M9 to 9.0.9 and 8.5.5 to 8.5.31.
>
> **CVSS Score:** 5.9
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2016-0762</summary>
> #### [CVE-2016-0762] Permissions, Privileges, and Access Controls
> The Realm implementations in Apache Tomcat versions 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0.0 to 7.0.70 and 6.0.0 to 6.0.45 did not process the supplied password if the supplied user name did not exist. This made a timing attack possible to determine valid user names. Note that the default configuration includes the LockOutRealm which makes exploitation of this vulnerability harder.
>
> **CVSS Score:** 5.9
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2008-0002</summary>
> #### [CVE-2008-0002] Apache Tomcat 6.0.0 through 6.0.15 processes parameters in the context of the wr...
> Apache Tomcat 6.0.0 through 6.0.15 processes parameters in the context of the wrong request when an exception occurs during parameter processing, which might allow remote attackers to obtain sensitive information, as demonstrated by disconnecting during this processing in order to trigger the exception.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2011-1088</summary>
> #### [CVE-2011-1088] Apache Tomcat 7.x before 7.0.10 does not follow ServletSecurity annotations, whi...
> Apache Tomcat 7.x before 7.0.10 does not follow ServletSecurity annotations, which allows remote attackers to bypass intended access restrictions via HTTP requests to a web application.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2011-1183</summary>
> #### [CVE-2011-1183] Apache Tomcat 7.0.11, when web.xml has no login configuration, does not follow s...
> Apache Tomcat 7.0.11, when web.xml has no login configuration, does not follow security constraints, which allows remote attackers to bypass intended access restrictions via HTTP requests to a meta-data complete web application. NOTE: this vulnerability exists because of an incorrect fix for CVE-2011-1088 and CVE-2011-1419.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2013-4286</summary>
> #### [CVE-2013-4286] Improper Input Validation
> Apache Tomcat before 6.0.39, 7.x before 7.0.47, and 8.x before 8.0.0-RC3, when an HTTP connector or AJP connector is used, does not properly handle certain inconsistent HTTP request headers, which allows remote attackers to trigger incorrect identification of a request's length and conduct request-smuggling attacks via (1) multiple Content-Length headers or (2) a Content-Length header and a "Transfer-Encoding: chunked" header. NOTE: this vulnerability exists because of an incomplete fix for CVE-2005-2090.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2011-1419</summary>
> #### [CVE-2011-1419] Apache Tomcat 7.x before 7.0.11, when web.xml has no security constraints, does ...
> Apache Tomcat 7.x before 7.0.11, when web.xml has no security constraints, does not follow ServletSecurity annotations, which allows remote attackers to bypass intended access restrictions via HTTP requests to a web application. NOTE: this vulnerability exists because of an incomplete fix for CVE-2011-1088.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:P/A:N
</details>
<details>
<summary>CVE-2009-2693</summary>
> #### [CVE-2009-2693] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Directory traversal vulnerability in Apache Tomcat 5.5.0 through 5.5.28 and 6.0.0 through 6.0.20 allows remote attackers to create or overwrite arbitrary files via a .. (dot dot) in an entry in a WAR file, as demonstrated by a ../../bin/catalina.bat entry.
>
> **CVSS Score:** 5.8
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:P
</details>
<details>
<summary>CVE-2021-33037</summary>
> #### [CVE-2021-33037] Apache Tomcat 10.0.0-M1 to 10.0.6, 9.0.0.M1 to 9.0.46 and 8.5.0 to 8.5.66 did no...
> Apache Tomcat 10.0.0-M1 to 10.0.6, 9.0.0.M1 to 9.0.46 and 8.5.0 to 8.5.66 did not correctly parse the HTTP transfer-encoding request header in some circumstances leading to the possibility to request smuggling when used with a reverse proxy. Specifically: - Tomcat incorrectly ignored the transfer encoding header if the client declared it would only accept an HTTP/1.0 response; - Tomcat honoured the identify encoding; and - Tomcat did not ensure that, if present, the chunked encoding was the final encoding.
>
> **CVSS Score:** 5.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:L/A:N
</details>
<details>
<summary>CVE-2015-5345</summary>
> #### [CVE-2015-5345] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> The Mapper component in Apache Tomcat 6.x before 6.0.45, 7.x before 7.0.68, 8.x before 8.0.30, and 9.x before 9.0.0.M2 processes redirects before considering security constraints and Filters, which allows remote attackers to determine the existence of a directory via a URL that lacks a trailing / (slash) character.
>
> **CVSS Score:** 5.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N
</details>
<details>
<summary>CVE-2017-15706</summary>
> #### [CVE-2017-15706] Improperly Implemented Security Check for Standard
> As part of the fix for bug 61201, the documentation for Apache Tomcat 9.0.0.M22 to 9.0.1, 8.5.16 to 8.5.23, 8.0.45 to 8.0.47 and 7.0.79 to 7.0.82 included an updated description of the search algorithm used by the CGI Servlet to identify which script to execute. The update was not correct. As a result, some scripts may have failed to execute as expected and other scripts may have been executed unexpectedly. Note that the behaviour of the CGI servlet has remained unchanged in this regard. It is only the documentation of the behaviour that was wrong and has been corrected.
>
> **CVSS Score:** 5.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:L/A:N
</details>
<details>
<summary>CVE-2016-6794</summary>
> #### [CVE-2016-6794] Information Exposure
> When a SecurityManager is configured, a web application's ability to read system properties should be controlled by the SecurityManager. In Apache Tomcat 9.0.0.M1 to 9.0.0.M9, 8.5.0 to 8.5.4, 8.0.0.RC1 to 8.0.36, 7.0.0 to 7.0.70, 6.0.0 to 6.0.45 the system property replacement feature for configuration files could be used by a malicious web application to bypass the SecurityManager and read system properties that should not be visible.
>
> **CVSS Score:** 5.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N
</details>
<details>
<summary>CVE-2001-0829</summary>
> #### [CVE-2001-0829] A cross-site scripting vulnerability in Apache Tomcat 3.2.1 allows a malicious w...
> A cross-site scripting vulnerability in Apache Tomcat 3.2.1 allows a malicious webmaster to embed Javascript in a request for a .JSP file, which causes the Javascript to be inserted into an error message.
>
> **CVSS Score:** 5.1
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2011-1184</summary>
> #### [CVE-2011-1184] Permissions, Privileges, and Access Controls
> The HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.12 does not have the expected countermeasures against replay attacks, which makes it easier for remote attackers to bypass intended access restrictions by sniffing the network for valid requests, related to lack of checking of nonce (aka server nonce) and nc (aka nonce-count or client nonce count) values.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-2370</summary>
> #### [CVE-2008-2370] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Apache Tomcat 4.1.0 through 4.1.37, 5.5.0 through 5.5.26, and 6.0.0 through 6.0.16, when a RequestDispatcher is used, performs path normalization before removing the query string from the URI, which allows remote attackers to conduct directory traversal attacks and read arbitrary files via a .. (dot dot) in a request parameter.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2011-4858</summary>
> #### [CVE-2011-4858] Resource Management Errors
> Apache Tomcat before 5.5.35, 6.x before 6.0.35, and 7.x before 7.0.23 computes hash values for form parameters without restricting the ability to trigger hash collisions predictably, which allows remote attackers to cause a denial of service (CPU consumption) by sending many crafted parameters.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2012-2733</summary>
> #### [CVE-2012-2733] Improper Input Validation
> java/org/apache/coyote/http11/InternalNioInputBuffer.java in the HTTP NIO connector in Apache Tomcat 6.x before 6.0.36 and 7.x before 7.0.28 does not properly restrict the request-header size, which allows remote attackers to cause a denial of service (memory consumption) via a large amount of header data.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2008-0128</summary>
> #### [CVE-2008-0128] null
> The SingleSignOn Valve (org.apache.catalina.authenticator.SingleSignOn) in Apache Tomcat before 5.5.21 does not set the secure flag for the JSESSIONIDSSO cookie in an https session, which can cause the cookie to be sent in http requests and make it easier for remote attackers to capture this cookie.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-2006</summary>
> #### [CVE-2002-2006] The default installation of Apache Tomcat 4.0 through 4.1 and 3.0 through 3.3.1 ...
> The default installation of Apache Tomcat 4.0 through 4.1 and 3.0 through 3.3.1 allows remote attackers to obtain the installation path and other sensitive system information via the (1) SnoopServlet or (2) TroubleShooter example servlets.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2011-0534</summary>
> #### [CVE-2011-0534] Resource Management Errors
> Apache Tomcat 7.0.0 through 7.0.6 and 6.0.0 through 6.0.30 does not enforce the maxHttpHeaderSize limit for requests involving the NIO HTTP connector, which allows remote attackers to cause a denial of service (OutOfMemoryError) via a crafted request.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2002-1895</summary>
> #### [CVE-2002-1895] The servlet engine in Jakarta Apache Tomcat 3.3 and 4.0.4, when using IIS and th...
> The servlet engine in Jakarta Apache Tomcat 3.3 and 4.0.4, when using IIS and the ajp1.3 connector, allows remote attackers to cause a denial of service (crash) via a large number of HTTP GET requests for an MS-DOS device such as AUX, LPT1, CON, or PRN.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2007-0450</summary>
> #### [CVE-2007-0450] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Directory traversal vulnerability in Apache HTTP Server and Tomcat 5.x before 5.5.22 and 6.x before 6.0.10, when using certain proxy modules (mod_proxy, mod_rewrite, mod_jk), allows remote attackers to read arbitrary files via a .. (dot dot) sequence with combinations of (1) "/" (slash), (2) "\" (backslash), and (3) URL-encoded backslash (%5C) characters in the URL, which are valid separators in Tomcat but not in Apache.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2001-0917</summary>
> #### [CVE-2001-0917] Jakarta Tomcat 4.0.1 allows remote attackers to reveal physical path information...
> Jakarta Tomcat 4.0.1 allows remote attackers to reveal physical path information by requesting a long URL with a .JSP extension.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2005-0808</summary>
> #### [CVE-2005-0808] Apache Tomcat before 5.x allows remote attackers to cause a denial of service (a...
> Apache Tomcat before 5.x allows remote attackers to cause a denial of service (application crash) via a crafted AJP12 packet to TCP port 8007.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2012-0022</summary>
> #### [CVE-2012-0022] Numeric Errors
> Apache Tomcat 5.5.x before 5.5.35, 6.x before 6.0.34, and 7.x before 7.0.23 uses an inefficient approach for handling parameters, which allows remote attackers to cause a denial of service (CPU consumption) via a request that contains many parameters and parameter values, a different vulnerability than CVE-2011-4858.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2000-0672</summary>
> #### [CVE-2000-0672] The default configuration of Jakarta Tomcat does not restrict access to the /adm...
> The default configuration of Jakarta Tomcat does not restrict access to the /admin context, which allows remote attackers to read arbitrary files by directly calling the administrative servlets to add a context for the root directory.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2012-5568</summary>
> #### [CVE-2012-5568] null
> Apache Tomcat through 7.0.x allows remote attackers to cause a denial of service (daemon outage) via partial HTTP requests, as demonstrated by Slowloris.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2011-1475</summary>
> #### [CVE-2011-1475] Improper Input Validation
> The HTTP BIO connector in Apache Tomcat 7.0.x before 7.0.12 does not properly handle HTTP pipelining, which allows remote attackers to read responses intended for other clients in opportunistic circumstances by examining the application data in HTTP packets, related to "a mix-up of responses for requests from different users."
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-2009</summary>
> #### [CVE-2002-2009] Apache Tomcat 4.0.1 allows remote attackers to obtain the web root path via HTTP...
> Apache Tomcat 4.0.1 allows remote attackers to obtain the web root path via HTTP requests for JSP files preceded by (1) +/, (2) >/, (3) </, and (4) %20/, which leaks the pathname in an error message.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2003-0043</summary>
> #### [CVE-2003-0043] Jakarta Tomcat before 3.3.1a, when used with JDK 1.3.1 or earlier, uses trusted ...
> Jakarta Tomcat before 3.3.1a, when used with JDK 1.3.1 or earlier, uses trusted privileges when processing the web.xml file, which could allow remote attackers to read portions of some files through the web.xml file.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2012-5886</summary>
> #### [CVE-2012-5886] Improper Authentication
> The HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.36, 6.x before 6.0.36, and 7.x before 7.0.30 caches information about the authenticated user within the session state, which makes it easier for remote attackers to bypass authentication via vectors related to the session ID.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2014-7810</summary>
> #### [CVE-2014-7810] Improper Access Control
> The Expression Language (EL) implementation in Apache Tomcat 6.x before 6.0.44, 7.x before 7.0.58, and 8.x before 8.0.16 does not properly consider the possibility of an accessible interface implemented by an inaccessible class, which allows attackers to bypass a SecurityManager protection mechanism via a web application that leverages use of incorrect privileges during EL evaluation.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2011-2729</summary>
> #### [CVE-2011-2729] Permissions, Privileges, and Access Controls
> native/unix/native/jsvc-unix.c in jsvc in the Daemon component 1.0.3 through 1.0.6 in Apache Commons, as used in Apache Tomcat 5.5.32 through 5.5.33, 6.0.30 through 6.0.32, and 7.0.x before 7.0.20 on Linux, does not drop capabilities, which allows remote attackers to bypass read permissions for files via a request to an application.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2003-0042</summary>
> #### [CVE-2003-0042] Jakarta Tomcat before 3.3.1a, when used with JDK 1.3.1 or earlier, allows remote...
> Jakarta Tomcat before 3.3.1a, when used with JDK 1.3.1 or earlier, allows remote attackers to list directories even with an index.html or other file present, or obtain unprocessed source code for a JSP file, via a URL containing a null character.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-0935</summary>
> #### [CVE-2002-0935] Apache Tomcat 4.0.3, and possibly other versions before 4.1.3 beta, allows remot...
> Apache Tomcat 4.0.3, and possibly other versions before 4.1.3 beta, allows remote attackers to cause a denial of service (resource exhaustion) via a large number of requests to the server with null characters, which causes the working threads to hang.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2003-0045</summary>
> #### [CVE-2003-0045] Jakarta Tomcat before 3.3.1a on certain Windows systems may allow remote attacke...
> Jakarta Tomcat before 3.3.1a on certain Windows systems may allow remote attackers to cause a denial of service (thread hang and resource consumption) via a request for a JSP page containing an MS-DOS device name, such as aux.jsp.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2007-5333</summary>
> #### [CVE-2007-5333] Information Exposure
> Apache Tomcat 6.0.0 through 6.0.14, 5.5.0 through 5.5.25, and 4.1.0 through 4.1.36 does not properly handle (1) double quote (") characters or (2) %5C (encoded backslash) sequences in a cookie value, which might cause sensitive information such as session IDs to be leaked to remote attackers and enable session hijacking attacks. NOTE: this issue exists because of an incomplete fix for CVE-2007-3385.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2014-0095</summary>
> #### [CVE-2014-0095] Improper Input Validation
> java/org/apache/coyote/ajp/AbstractAjpProcessor.java in Apache Tomcat 8.x before 8.0.4 allows remote attackers to cause a denial of service (thread consumption) by using a "Content-Length: 0" AJP request to trigger a hang in request processing.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2012-3544</summary>
> #### [CVE-2012-3544] Improper Input Validation
> Apache Tomcat 6.x before 6.0.37 and 7.x before 7.0.30 does not properly handle chunk extensions in chunked transfer coding, which allows remote attackers to cause a denial of service by streaming data.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2005-4703</summary>
> #### [CVE-2005-4703] Apache Tomcat 4.0.3, when running on Windows, allows remote attackers to obtain ...
> Apache Tomcat 4.0.3, when running on Windows, allows remote attackers to obtain sensitive information via a request for a file that contains an MS-DOS device name such as lpt9, which leaks the pathname in an error message, as demonstrated by lpt9.xtp using Nikto.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-0936</summary>
> #### [CVE-2002-0936] The Java Server Pages (JSP) engine in Tomcat allows web page owners to cause a d...
> The Java Server Pages (JSP) engine in Tomcat allows web page owners to cause a denial of service (engine crash) on the web server via a JSP page that calls WPrinterJob().pageSetup(null,null).
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2006-3835</summary>
> #### [CVE-2006-3835] Apache Tomcat 5 before 5.5.17 allows remote attackers to list directories via a ...
> Apache Tomcat 5 before 5.5.17 allows remote attackers to list directories via a semicolon (;) preceding a filename with a mapped extension, as demonstrated by URLs ending with /;index.jsp and /;help.do.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-2007</summary>
> #### [CVE-2002-2007] The default installations of Apache Tomcat 3.2.3 and 3.2.4 allows remote attacke...
> The default installations of Apache Tomcat 3.2.3 and 3.2.4 allows remote attackers to obtain sensitive system information such as directory listings and web root path, via erroneous HTTP requests for Java Server Pages (JSP) in the (1) test/jsp, (2) samples/jsp and (3) examples/jsp directories, or the (4) test/realPath.jsp servlet, which leaks pathnames in error messages.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2002-2008</summary>
> #### [CVE-2002-2008] Apache Tomcat 4.0.3 for Windows allows remote attackers to obtain the web root p...
> Apache Tomcat 4.0.3 for Windows allows remote attackers to obtain the web root path via an HTTP request for a resource that does not exist, such as lpt9, which leaks the information in an error message.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2001-0590</summary>
> #### [CVE-2001-0590] Apache Software Foundation Tomcat Servlet prior to 3.2.2 allows a remote attacke...
> Apache Software Foundation Tomcat Servlet prior to 3.2.2 allows a remote attacker to read the source code to arbitrary 'jsp' files via a malformed URL request which does not end with an HTTP protocol specification (i.e. HTTP/1.0).
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2009-0033</summary>
> #### [CVE-2009-0033] Improper Input Validation
> Apache Tomcat 4.1.0 through 4.1.39, 5.5.0 through 5.5.27, and 6.0.0 through 6.0.18, when the Java AJP connector and mod_jk load balancing are used, allows remote attackers to cause a denial of service (application outage) via a crafted request with invalid headers, related to temporary blocking of connectors that have encountered errors, as demonstrated by an error involving a malformed HTTP Host header.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2012-5887</summary>
> #### [CVE-2012-5887] Improper Authentication
> The HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.36, 6.x before 6.0.36, and 7.x before 7.0.30 does not properly check for stale nonce values in conjunction with enforcement of proper credentials, which makes it easier for remote attackers to bypass intended access restrictions by sniffing the network for valid requests.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2002-1148</summary>
> #### [CVE-2002-1148] The default servlet (org.apache.catalina.servlets.DefaultServlet) in Tomcat 4.0....
> The default servlet (org.apache.catalina.servlets.DefaultServlet) in Tomcat 4.0.4 and 4.1.10 and earlier allows remote attackers to read source code for server files via a direct request to the servlet.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2003-0866</summary>
> #### [CVE-2003-0866] The Catalina org.apache.catalina.connector.http package in Tomcat 4.0.x up to 4....
> The Catalina org.apache.catalina.connector.http package in Tomcat 4.0.x up to 4.0.3 allows remote attackers to cause a denial of service via several requests that do not follow the HTTP protocol, which causes Tomcat to reject later requests.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2011-3375</summary>
> #### [CVE-2011-3375] Information Exposure
> Apache Tomcat 6.0.30 through 6.0.33 and 7.x before 7.0.22 does not properly perform certain caching and recycling operations involving request objects, which allows remote attackers to obtain unintended read access to IP address and HTTP header information in opportunistic circumstances by reading TCP data.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2014-0075</summary>
> #### [CVE-2014-0075] Numeric Errors
> Integer overflow in the parseChunkHeader function in java/org/apache/coyote/http11/filters/ChunkedInputFilter.java in Apache Tomcat before 6.0.40, 7.x before 7.0.53, and 8.x before 8.0.4 allows remote attackers to cause a denial of service (resource consumption) via a malformed chunk size in chunked transfer coding of a request during the streaming of data.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2011-5062</summary>
> #### [CVE-2011-5062] Permissions, Privileges, and Access Controls
> The HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.12 does not check qop values, which might allow remote attackers to bypass intended integrity-protection requirements via a qop=auth value, a different vulnerability than CVE-2011-1184.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2000-1210</summary>
> #### [CVE-2000-1210] Directory traversal vulnerability in source.jsp of Apache Tomcat before 3.1 allo...
> Directory traversal vulnerability in source.jsp of Apache Tomcat before 3.1 allows remote attackers to read arbitrary files via a .. (dot dot) in the argument to source.jsp.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-5515</summary>
> #### [CVE-2008-5515] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Apache Tomcat 4.1.0 through 4.1.39, 5.5.0 through 5.5.27, 6.0.0 through 6.0.18, and possibly earlier versions normalizes the target pathname before filtering the query string when using the RequestDispatcher method, which allows remote attackers to bypass intended access restrictions and conduct directory traversal attacks via .. (dot dot) sequences and the WEB-INF directory in a Request.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2005-3510</summary>
> #### [CVE-2005-3510] Apache Tomcat 5.5.0 to 5.5.11 allows remote attackers to cause a denial of servi...
> Apache Tomcat 5.5.0 to 5.5.11 allows remote attackers to cause a denial of service (CPU consumption) via a large number of simultaneous requests to list a web directory that has a large number of files.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2012-5885</summary>
> #### [CVE-2012-5885] Permissions, Privileges, and Access Controls
> The replay-countermeasure functionality in the HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.36, 6.x before 6.0.36, and 7.x before 7.0.30 tracks cnonce (aka client nonce) values instead of nonce (aka server nonce) and nc (aka nonce-count) values, which makes it easier for remote attackers to bypass intended access restrictions by sniffing the network for valid requests, a different vulnerability than CVE-2011-1184.
>
> **CVSS Score:** 5
>
> **CVSS Vector:** AV:N/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2019-17569</summary>
> #### [CVE-2019-17569] The refactoring present in Apache Tomcat 9.0.28 to 9.0.30, 8.5.48 to 8.5.50 and ...
> The refactoring present in Apache Tomcat 9.0.28 to 9.0.30, 8.5.48 to 8.5.50 and 7.0.98 to 7.0.99 introduced a regression. The result of the regression was that invalid Transfer-Encoding headers were incorrectly processed leading to a possibility of HTTP Request Smuggling if Tomcat was located behind a reverse proxy that incorrectly handled the invalid Transfer-Encoding header in a particular manner. Such a reverse proxy is considered unlikely.
>
> **CVSS Score:** 4.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:L/A:N
</details>
<details>
<summary>CVE-2020-1935</summary>
> #### [CVE-2020-1935] In Apache Tomcat 9.0.0.M1 to 9.0.30, 8.5.0 to 8.5.50 and 7.0.0 to 7.0.99 the HTT...
> In Apache Tomcat 9.0.0.M1 to 9.0.30, 8.5.0 to 8.5.50 and 7.0.0 to 7.0.99 the HTTP header parsing code used an approach to end-of-line parsing that allowed some invalid HTTP headers to be parsed as valid. This led to a possibility of HTTP Request Smuggling if Tomcat was located behind a reverse proxy that incorrectly handled the invalid Transfer-Encoding header in a particular manner. Such a reverse proxy is considered unlikely.
>
> **CVSS Score:** 4.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:L/A:N
</details>
<details>
<summary>CVE-2011-2481</summary>
> #### [CVE-2011-2481] Apache Tomcat 7.0.x before 7.0.17 permits web applications to replace an XML par...
> Apache Tomcat 7.0.x before 7.0.17 permits web applications to replace an XML parser used for other web applications, which allows local users to read or modify the (1) web.xml, (2) context.xml, or (3) tld files of arbitrary web applications via a crafted application that is loaded earlier than the target application. NOTE: this vulnerability exists because of a CVE-2009-0783 regression.
>
> **CVSS Score:** 4.6
>
> **CVSS Vector:** AV:L/AC:L/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2011-3376</summary>
> #### [CVE-2011-3376] Permissions, Privileges, and Access Controls
> org/apache/catalina/core/DefaultInstanceManager.java in Apache Tomcat 7.x before 7.0.22 does not properly restrict ContainerServlets in the Manager application, which allows local users to gain privileges by using an untrusted web application to access the Manager application's functionality.
>
> **CVSS Score:** 4.4
>
> **CVSS Vector:** AV:L/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2011-2526</summary>
> #### [CVE-2011-2526] Improper Input Validation
> Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.19, when sendfile is enabled for the HTTP APR or HTTP NIO connector, does not validate certain request attributes, which allows local users to bypass intended file access restrictions or cause a denial of service (infinite loop or JVM crash) by leveraging an untrusted web application.
>
> **CVSS Score:** 4.4
>
> **CVSS Vector:** AV:L/AC:M/Au:N/C:P/I:P/A:P
</details>
<details>
<summary>CVE-2011-5064</summary>
> #### [CVE-2011-5064] Cryptographic Issues
> DigestAuthenticator.java in the HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.12 uses Catalina as the hard-coded server secret (aka private key), which makes it easier for remote attackers to bypass cryptographic protection mechanisms by leveraging knowledge of this string, a different vulnerability than CVE-2011-1184.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-1947</summary>
> #### [CVE-2008-1947] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in Apache Tomcat 5.5.9 through 5.5.26 and 6.0.0 through 6.0.16 allows remote attackers to inject arbitrary web script or HTML via the name parameter (aka the hostname attribute) to host-manager/html/add.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2016-0706</summary>
> #### [CVE-2016-0706] Information Exposure
> Apache Tomcat 6.x before 6.0.45, 7.x before 7.0.68, 8.x before 8.0.31, and 9.x before 9.0.0.M2 does not place org.apache.catalina.manager.StatusManagerServlet on the org/apache/catalina/core/RestrictedServlets.properties list, which allows remote authenticated users to bypass intended SecurityManager restrictions and read arbitrary HTTP requests, and consequently discover session ID values, via a crafted web application.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N
</details>
<details>
<summary>CVE-2014-0119</summary>
> #### [CVE-2014-0119] Permissions, Privileges, and Access Controls
> Apache Tomcat before 6.0.40, 7.x before 7.0.54, and 8.x before 8.0.6 does not properly constrain the class loader that accesses the XML parser used with an XSLT stylesheet, which allows remote attackers to (1) read arbitrary files via a crafted web application that provides an XML external entity declaration in conjunction with an entity reference, related to an XML External Entity (XXE) issue, or (2) read files associated with different web applications on a single Tomcat instance via a crafted web application.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2006-7195</summary>
> #### [CVE-2006-7195] Cross-site scripting (XSS) vulnerability in implicit-objects.jsp in Apache Tomca...
> Cross-site scripting (XSS) vulnerability in implicit-objects.jsp in Apache Tomcat 5.0.0 through 5.0.30 and 5.5.0 through 5.5.17 allows remote attackers to inject arbitrary web script or HTML via certain header values.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2013-4322</summary>
> #### [CVE-2013-4322] Improper Input Validation
> Apache Tomcat before 6.0.39, 7.x before 7.0.50, and 8.x before 8.0.0-RC10 processes chunked transfer coding without properly handling (1) a large total amount of chunked data or (2) whitespace characters in an HTTP header value within a trailer field, which allows remote attackers to cause a denial of service by streaming data. NOTE: this vulnerability exists because of an incomplete fix for CVE-2012-3544.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2011-1582</summary>
> #### [CVE-2011-1582] Permissions, Privileges, and Access Controls
> Apache Tomcat 7.0.12 and 7.0.13 processes the first request to a servlet without following security constraints that have been configured through annotations, which allows remote attackers to bypass intended access restrictions via HTTP requests. NOTE: this vulnerability exists because of an incomplete fix for CVE-2011-1088, CVE-2011-1183, and CVE-2011-1419.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2010-4172</summary>
> #### [CVE-2010-4172] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Multiple cross-site scripting (XSS) vulnerabilities in the Manager application in Apache Tomcat 6.0.12 through 6.0.29 and 7.0.0 through 7.0.4 allow remote attackers to inject arbitrary web script or HTML via the (1) orderBy or (2) sort parameter to sessionsList.jsp, or unspecified input to (3) sessionDetail.jsp or (4) java/org/apache/catalina/manager/JspHelper.java, related to use of untrusted web applications.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-2449</summary>
> #### [CVE-2007-2449] Multiple cross-site scripting (XSS) vulnerabilities in certain JSP files in the ...
> Multiple cross-site scripting (XSS) vulnerabilities in certain JSP files in the examples web application in Apache Tomcat 4.0.0 through 4.0.6, 4.1.0 through 4.1.36, 5.0.0 through 5.0.30, 5.5.0 through 5.5.24, and 6.0.0 through 6.0.13 allow remote attackers to inject arbitrary web script or HTML via the portion of the URI after the ';' character, as demonstrated by a URI containing a "snp/snoop.jsp;" sequence.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2012-4431</summary>
> #### [CVE-2012-4431] Permissions, Privileges, and Access Controls
> org/apache/catalina/filters/CsrfPreventionFilter.java in Apache Tomcat 6.x before 6.0.36 and 7.x before 7.0.32 allows remote attackers to bypass the cross-site request forgery (CSRF) protection mechanism via a request that lacks a session identifier.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-3385</summary>
> #### [CVE-2007-3385] Information Exposure
> Apache Tomcat 6.0.0 to 6.0.13, 5.5.0 to 5.5.24, 5.0.0 to 5.0.30, 4.1.0 to 4.1.36, and 3.3 to 3.3.2 does not properly handle the \" character sequence in a cookie value, which might cause sensitive information such as session IDs to be leaked to remote attackers and enable session hijacking attacks.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2006-7196</summary>
> #### [CVE-2006-7196] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in the calendar application example in Apache Tomcat 4.0.0 through 4.0.6, 4.1.0 through 4.1.31, 5.0.0 through 5.0.30, and 5.5.0 through 5.5.15 allows remote attackers to inject arbitrary web script or HTML via the time parameter to cal2.jsp and possibly unspecified other vectors. NOTE: this may be related to CVE-2006-0254.1.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-3384</summary>
> #### [CVE-2007-3384] Multiple cross-site scripting (XSS) vulnerabilities in examples/servlet/CookieEx...
> Multiple cross-site scripting (XSS) vulnerabilities in examples/servlet/CookieExample in Apache Tomcat 3.3 through 3.3.2 allow remote attackers to inject arbitrary web script or HTML via the (1) Name or (2) Value field, related to error messages.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2011-5063</summary>
> #### [CVE-2011-5063] Improper Authentication
> The HTTP Digest Access Authentication implementation in Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.12 does not check realm values, which might allow remote attackers to bypass intended access restrictions by leveraging the availability of a protection space with weaker authentication or authorization requirements, a different vulnerability than CVE-2011-1184.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2014-0096</summary>
> #### [CVE-2014-0096] Permissions, Privileges, and Access Controls
> java/org/apache/catalina/servlets/DefaultServlet.java in the default servlet in Apache Tomcat before 6.0.40, 7.x before 7.0.53, and 8.x before 8.0.4 does not properly restrict XSLT stylesheets, which allows remote attackers to bypass security-manager restrictions and read arbitrary files via a crafted web application that provides an XML external entity declaration in conjunction with an entity reference, related to an XML External Entity (XXE) issue.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-1232</summary>
> #### [CVE-2008-1232] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in Apache Tomcat 4.1.0 through 4.1.37, 5.5.0 through 5.5.26, and 6.0.0 through 6.0.16 allows remote attackers to inject arbitrary web script or HTML via a crafted string that is used in the message argument to the HttpServletResponse.sendError method.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-3382</summary>
> #### [CVE-2007-3382] Information Exposure
> Apache Tomcat 6.0.0 to 6.0.13, 5.5.0 to 5.5.24, 5.0.0 to 5.0.30, 4.1.0 to 4.1.36, and 3.3 to 3.3.2 treats single quotes ("'") as delimiters in cookies, which might cause sensitive information such as session IDs to be leaked and allow remote attackers to conduct session hijacking attacks.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2017-7674</summary>
> #### [CVE-2017-7674] Insufficient Verification of Data Authenticity
> The CORS Filter in Apache Tomcat 9.0.0.M1 to 9.0.0.M21, 8.5.0 to 8.5.15, 8.0.0.RC1 to 8.0.44 and 7.0.41 to 7.0.78 did not add an HTTP Vary header indicating that the response varies depending on Origin. This permitted client and server side cache poisoning in some circumstances.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:L/A:N
</details>
<details>
<summary>CVE-2007-4724</summary>
> #### [CVE-2007-4724] Cross-Site Request Forgery (CSRF)
> Cross-site request forgery (CSRF) vulnerability in cal2.jsp in the calendar examples application in Apache Tomcat 4.1.31 allows remote attackers to add events as arbitrary users via the time and description parameters.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2015-5174</summary>
> #### [CVE-2015-5174] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Directory traversal vulnerability in RequestUtil.java in Apache Tomcat 6.x before 6.0.45, 7.x before 7.0.65, and 8.x before 8.0.27 allows remote authenticated users to bypass intended SecurityManager restrictions and list a parent directory via a /.. (slash dot dot) in a pathname used by a web application in a getResource, getResourceAsStream, or getResourcePaths call, as demonstrated by the $CATALINA_BASE/webapps directory.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N
</details>
<details>
<summary>CVE-2009-2901</summary>
> #### [CVE-2009-2901] Permissions, Privileges, and Access Controls
> The autodeployment process in Apache Tomcat 5.5.0 through 5.5.28 and 6.0.0 through 6.0.20, when autoDeploy is enabled, deploys appBase files that remain from a failed undeploy, which might allow remote attackers to bypass intended authentication requirements via HTTP requests.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2007-6286</summary>
> #### [CVE-2007-6286] Apache Tomcat 5.5.11 through 5.5.25 and 6.0.0 through 6.0.15, when the native AP...
> Apache Tomcat 5.5.11 through 5.5.25 and 6.0.0 through 6.0.15, when the native APR connector is used, does not properly handle an empty request to the SSL port, which allows remote attackers to trigger handling of "a duplicate copy of one of the recent requests," as demonstrated by using netcat to send the empty request.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2014-0033</summary>
> #### [CVE-2014-0033] Improper Input Validation
> org/apache/catalina/connector/CoyoteAdapter.java in Apache Tomcat 6.0.33 through 6.0.37 does not consider the disableURLRewriting setting when handling a session ID in a URL, which allows remote attackers to conduct session fixation attacks via a crafted URL.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2009-2902</summary>
> #### [CVE-2009-2902] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Directory traversal vulnerability in Apache Tomcat 5.5.0 through 5.5.28 and 6.0.0 through 6.0.20 allows remote attackers to delete work-directory files via directory traversal sequences in a WAR filename, as demonstrated by the ...war filename.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2009-2696</summary>
> #### [CVE-2009-2696] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in jsp/cal/cal2.jsp in the calendar application in the examples web application in Apache Tomcat on Red Hat Enterprise Linux 5, Desktop Workstation 5, and Linux Desktop 5 allows remote attackers to inject arbitrary web script or HTML via the time parameter, related to "invalid HTML." NOTE: this is due to a missing fix for CVE-2009-0781.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2005-2090</summary>
> #### [CVE-2005-2090] Jakarta Tomcat 5.0.19 (Coyote/1.1) and Tomcat 4.1.24 (Coyote/1.0) allows remote ...
> Jakarta Tomcat 5.0.19 (Coyote/1.1) and Tomcat 4.1.24 (Coyote/1.0) allows remote attackers to poison the web cache, bypass web application firewall protection, and conduct XSS attacks via an HTTP request with both a "Transfer-Encoding: chunked" header and a Content-Length header, which causes Tomcat to incorrectly handle and forward the body of the request in a way that causes the receiving server to process it as a separate HTTP request, aka "HTTP Request Smuggling."
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2005-4838</summary>
> #### [CVE-2005-4838] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Multiple cross-site scripting (XSS) vulnerabilities in the example web applications for Jakarta Tomcat 5.5.6 and earlier allow remote attackers to inject arbitrary web script or HTML via (1) el/functions.jsp, (2) el/implicit-objects.jsp, and (3) jspx/textRotate.jspx in examples/jsp2/, as demonstrated via script in a request to snp/snoop.jsp. NOTE: other XSS issues in the manager were simultaneously reported, but these require admin access and do not cross privilege boundaries.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2008-3271</summary>
> #### [CVE-2008-3271] Permissions, Privileges, and Access Controls
> Apache Tomcat 5.5.0 and 4.1.0 through 4.1.31 allows remote attackers to bypass an IP address restriction and obtain sensitive information via a request that is processed concurrently with another request but in a different thread, leading to an instance-variable overwrite associated with a "synchronization problem" and lack of thread safety, and related to RemoteFilterValve, RemoteAddrValve, and RemoteHostValve.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2020-13943</summary>
> #### [CVE-2020-13943] If an HTTP/2 client connecting to Apache Tomcat 10.0.0-M1 to 10.0.0-M7, 9.0.0.M1...
> If an HTTP/2 client connecting to Apache Tomcat 10.0.0-M1 to 10.0.0-M7, 9.0.0.M1 to 9.0.37 or 8.5.0 to 8.5.57 exceeded the agreed maximum number of concurrent streams for a connection (in violation of the HTTP/2 protocol), it was possible that a subsequent request made on that connection could contain HTTP headers - including HTTP/2 pseudo headers - from a previous request rather than the intended headers. This could lead to users seeing responses for unexpected resources.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N
</details>
<details>
<summary>CVE-2011-0013</summary>
> #### [CVE-2011-0013] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Multiple cross-site scripting (XSS) vulnerabilities in the HTML Manager Interface in Apache Tomcat 5.5 before 5.5.32, 6.0 before 6.0.30, and 7.0 before 7.0.6 allow remote attackers to inject arbitrary web script or HTML, as demonstrated via the display-name tag.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-3383</summary>
> #### [CVE-2007-3383] Cross-site scripting (XSS) vulnerability in SendMailServlet in the examples web ...
> Cross-site scripting (XSS) vulnerability in SendMailServlet in the examples web application (examples/jsp/mail/sendmail.jsp) in Apache Tomcat 4.0.0 through 4.0.6 and 4.1.0 through 4.1.36 allows remote attackers to inject arbitrary web script or HTML via the From field and possibly other fields, related to generation of error messages.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2009-0580</summary>
> #### [CVE-2009-0580] Information Exposure
> Apache Tomcat 4.1.0 through 4.1.39, 5.5.0 through 5.5.27, and 6.0.0 through 6.0.18, when FORM authentication is used, allows remote attackers to enumerate valid usernames via requests to /j_security_check with malformed URL encoding of passwords, related to improper error checking in the (1) MemoryRealm, (2) DataSourceRealm, and (3) JDBCRealm authentication realms, as demonstrated by a % (percent) value for the j_password parameter.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2012-3546</summary>
> #### [CVE-2012-3546] Permissions, Privileges, and Access Controls
> org/apache/catalina/realm/RealmBase.java in Apache Tomcat 6.x before 6.0.36 and 7.x before 7.0.30, when FORM authentication is used, allows remote attackers to bypass security-constraint checks by leveraging a previous setUserPrincipal call and then placing /j_security_check at the end of a URI.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2013-4590</summary>
> #### [CVE-2013-4590] Information Exposure
> Apache Tomcat before 6.0.39, 7.x before 7.0.50, and 8.x before 8.0.0-RC10 allows attackers to obtain "Tomcat internals" information by leveraging the presence of an untrusted web application with a context.xml, web.xml, *.jspx, *.tagx, or *.tld XML document containing an external entity declaration in conjunction with an entity reference, related to an XML External Entity (XXE) issue.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2014-0099</summary>
> #### [CVE-2014-0099] Numeric Errors
> Integer overflow in java/org/apache/tomcat/util/buf/Ascii.java in Apache Tomcat before 6.0.40, 7.x before 7.0.53, and 8.x before 8.0.4, when operated behind a reverse proxy, allows remote attackers to conduct HTTP request smuggling attacks via a crafted Content-Length HTTP header.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2018-11784</summary>
> #### [CVE-2018-11784] URL Redirection to Untrusted Site ("Open Redirect")
> When the default servlet in Apache Tomcat versions 9.0.0.M1 to 9.0.11, 8.5.0 to 8.5.33 and 7.0.23 to 7.0.90 returned a redirect to a directory (e.g. redirecting to '/foo/' when the user requested '/foo') a specially crafted URL could be used to cause the redirect to be generated to any URI of the attackers choice.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:L/A:N
</details>
<details>
<summary>CVE-2007-3386</summary>
> #### [CVE-2007-3386] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in the Host Manager Servlet for Apache Tomcat 6.0.0 to 6.0.13 and 5.5.0 to 5.5.24 allows remote attackers to inject arbitrary HTML and web script via crafted requests, as demonstrated using the aliases parameter to an html/add action.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2009-0781</summary>
> #### [CVE-2009-0781] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in jsp/cal/cal2.jsp in the calendar application in the examples web application in Apache Tomcat 4.1.0 through 4.1.39, 5.5.0 through 5.5.27, and 6.0.0 through 6.0.18 allows remote attackers to inject arbitrary web script or HTML via the time parameter, related to "invalid HTML."
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2007-1355</summary>
> #### [CVE-2007-1355] Multiple cross-site scripting (XSS) vulnerabilities in the appdev/sample/web/hel...
> Multiple cross-site scripting (XSS) vulnerabilities in the appdev/sample/web/hello.jsp example application in Tomcat 4.0.0 through 4.0.6, 4.1.0 through 4.1.36, 5.0.0 through 5.0.30, 5.5.0 through 5.5.23, and 6.0.0 through 6.0.10 allow remote attackers to inject arbitrary web script or HTML via the test parameter and unspecified vectors.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2008-2938</summary>
> #### [CVE-2008-2938] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Directory traversal vulnerability in Apache Tomcat 4.1.0 through 4.1.37, 5.5.0 through 5.5.26, and 6.0.0 through 6.0.16, when allowLinking and UTF-8 are enabled, allows remote attackers to read arbitrary files via encoded directory traversal sequences in the URI, a different vulnerability than CVE-2008-2370. NOTE: versions earlier than 6.0.18 were reported affected, but the vendor advisory lists 6.0.16 as the last affected version.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** AV:N/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2009-0783</summary>
> #### [CVE-2009-0783] Information Exposure
> Apache Tomcat 4.1.0 through 4.1.39, 5.5.0 through 5.5.27, and 6.0.0 through 6.0.18 permits web applications to replace an XML parser used for other web applications, which allows local users to read or modify the (1) web.xml, (2) context.xml, or (3) tld files of arbitrary web applications via a crafted application that is loaded earlier than the target application.
>
> **CVSS Score:** 4.2
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:L/PR:H/UI:N/S:U/C:L/I:L/A:L
</details>
</ul>
</details>
<details>
<summary><b>MODERATE Vulnerabilities (13)</b></summary><br/>
<ul>
<details>
<summary>CVE-2007-5461</summary>
> #### [CVE-2007-5461] Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
> Absolute path traversal vulnerability in Apache Tomcat 4.0.0 through 4.0.6, 4.1.0, 5.0.0, 5.5.0 through 5.5.25, and 6.0.0 through 6.0.14, under certain configurations, allows remote authenticated users to read arbitrary files via a WebDAV write request that specifies an entity with a SYSTEM tag.
>
> **CVSS Score:** 3.5
>
> **CVSS Vector:** AV:N/AC:M/Au:S/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2007-2450</summary>
> #### [CVE-2007-2450] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Multiple cross-site scripting (XSS) vulnerabilities in the (1) Manager and (2) Host Manager web applications in Apache Tomcat 4.0.0 through 4.0.6, 4.1.0 through 4.1.36, 5.0.0 through 5.0.30, 5.5.0 through 5.5.24, and 6.0.0 through 6.0.13 allow remote authenticated users to inject arbitrary web script or HTML via a parameter name to manager/html/upload, and other unspecified vectors.
>
> **CVSS Score:** 3.5
>
> **CVSS Vector:** AV:N/AC:M/Au:S/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2005-3164</summary>
> #### [CVE-2005-3164] Information Exposure
> The AJP connector in Apache Tomcat 4.0.1 through 4.0.6 and 4.1.0 through 4.1.36, as used in Hitachi Cosminexus Application Server and standalone, does not properly handle when a connection is broken before request body data is sent in a POST request, which can lead to an information leak when "unsuitable request body data" is used for a different request, possibly related to Java Servlet pages.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-5519</summary>
> #### [CVE-2008-5519] Information Exposure
> The JK Connector (aka mod_jk) 1.2.0 through 1.2.26 in Apache Tomcat allows remote attackers to obtain sensitive information via an arbitrary request from an HTTP client, in opportunistic circumstances involving (1) a request from a different client that included a Content-Length header but no POST data or (2) a rapid series of requests, related to noncompliance with the AJP protocol's requirements for requests containing Content-Length headers.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2007-1358</summary>
> #### [CVE-2007-1358] Improper Neutralization of Input During Web Page Generation ("Cross-site Scripting")
> Cross-site scripting (XSS) vulnerability in certain applications using Apache Tomcat 4.0.0 through 4.0.6 and 4.1.0 through 4.1.34 allows remote attackers to inject arbitrary web script or HTML via crafted "Accept-Language headers that do not conform to RFC 2616".
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:N/I:P/A:N
</details>
<details>
<summary>CVE-2013-2071</summary>
> #### [CVE-2013-2071] Information Exposure
> java/org/apache/catalina/core/AsyncContextImpl.java in Apache Tomcat 7.x before 7.0.40 does not properly handle the throwing of a RuntimeException in an AsyncListener in an application, which allows context-dependent attackers to obtain sensitive request information intended for other applications in opportunistic circumstances via an application that records the requests that it processes.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2008-4308</summary>
> #### [CVE-2008-4308] Information Exposure
> The doRead method in Apache Tomcat 4.1.32 through 4.1.34 and 5.5.10 through 5.5.20 does not return a -1 to indicate when a certain error condition has occurred, which can cause Tomcat to send POST content from one request to a different request.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2010-1157</summary>
> #### [CVE-2010-1157] Information Exposure
> Apache Tomcat 5.5.0 through 5.5.29 and 6.0.0 through 6.0.26 might allow remote attackers to discover the server's hostname or IP address by sending a request for a resource that requires (1) BASIC or (2) DIGEST authentication, and then reading the realm field in the WWW-Authenticate header in the reply.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2012-4534</summary>
> #### [CVE-2012-4534] Resource Management Errors
> org/apache/tomcat/util/net/NioEndpoint.java in Apache Tomcat 6.x before 6.0.36 and 7.x before 7.0.28, when the NIO connector is used in conjunction with sendfile and HTTPS, allows remote attackers to cause a denial of service (infinite loop) by terminating the connection during the reading of a response.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:N/I:N/A:P
</details>
<details>
<summary>CVE-2007-1858</summary>
> #### [CVE-2007-1858] The default SSL cipher configuration in Apache Tomcat 4.1.28 through 4.1.31, 5.0...
> The default SSL cipher configuration in Apache Tomcat 4.1.28 through 4.1.31, 5.0.0 through 5.0.30, and 5.5.0 through 5.5.17 uses certain insecure ciphers, including the anonymous cipher, which allows remote attackers to obtain sensitive information or have other, unspecified impacts.
>
> **CVSS Score:** 2.6
>
> **CVSS Vector:** AV:N/AC:H/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2013-0346</summary>
> #### [CVE-2013-0346] Permissions, Privileges, and Access Controls
> ** DISPUTED ** Apache Tomcat 7.x uses world-readable permissions for the log directory and its files, which might allow local users to obtain sensitive information by reading a file. NOTE: One Tomcat distributor has stated "The tomcat log directory does not contain any sensitive information."
>
> **CVSS Score:** 2.1
>
> **CVSS Vector:** AV:L/AC:L/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2011-2204</summary>
> #### [CVE-2011-2204] Information Exposure
> Apache Tomcat 5.5.x before 5.5.34, 6.x before 6.0.33, and 7.x before 7.0.17, when the MemoryUserDatabase is used, creates log entries containing passwords upon encountering errors in JMX user creation, which allows local users to obtain sensitive information by reading a log file.
>
> **CVSS Score:** 1.9
>
> **CVSS Vector:** AV:L/AC:M/Au:N/C:P/I:N/A:N
</details>
<details>
<summary>CVE-2010-3718</summary>
> #### [CVE-2010-3718] Apache Tomcat 7.0.0 through 7.0.3, 6.0.x, and 5.5.x, when running within a Secur...
> Apache Tomcat 7.0.0 through 7.0.3, 6.0.x, and 5.5.x, when running within a SecurityManager, does not make the ServletContext attribute read-only, which allows local web applications to read or write files outside of the intended working directory, as demonstrated using a directory traversal attack.
>
> **CVSS Score:** 1.2
>
> **CVSS Vector:** AV:L/AC:H/Au:N/C:N/I:P/A:N
</details>
</ul>
</details>
</ul>
</details>
</ul>
</details>
(at-me [in a reply](https://help.sonatype.com/lift/talking-to-lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714649231
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBMetricsDAOTest.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.skywalking.oap.server.core.analysis.DownSampling;
+import org.apache.skywalking.oap.server.core.analysis.FunctionCategory;
+import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic;
+import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.model.ExtraQueryIndex;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBStorageConfig;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+import static org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClientTest.retrieval;
+
+public class IoTDBMetricsDAOTest {
Review comment:
Could we only test these DAOs in e2e rather than in Maven IT? It reduces resource costs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] jixuan1989 commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-925784401
> SkyWalking is building dev image automatically for every default branch
commit, using commitid as tag.
Thanks, I will have a try to enable the feature of "Packages" :D
吴晟 Wu Sheng ***@***.***> 于2021年9月23日周四 下午6:58写道：
> I will package a IoTDB dev docker image later.
>
> SkyWalking is building dev image automatically for every default branch
> commit, using commitid as tag.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/skywalking/pull/7766#issuecomment-925707171>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHZOVRYR3NW54KXUOM7TLTUDMB5VANCNFSM5EQWDD3Q>
> .
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-945743390
You need to use `infra-e2e` tool when you resolve conflicts. The v1 e2e framework is being removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713760341
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,475 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.exception.NullFieldException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String GROUP_IDX = "service_group";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) throws IOException {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ public final String getStorageGroup() {
+ return storageGroup;
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize(), config.isRpcCompression());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getTimeseriesList(), request.getTimeseriesTypes(), request.getTimeseriesValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getTimeseriesList());
+ typesList.add(request.getTimeseriesTypes());
+ valuesList.add(request.getTimeseriesValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(modelName);
+ if (!sessionPool.checkTimeseriesExists(devicePath.toString())) {
+ return storageDataList;
+ }
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
+ }
+ if (map.containsKey(IoTDBClient.NODE_TYPE_IDX)) {
+ String nodeType = (String) map.get(IoTDBClient.NODE_TYPE_IDX);
+ map.put(IoTDBClient.NODE_TYPE_IDX, Integer.valueOf(nodeType));
+ }
+ if (modelName.equals(BrowserErrorLogRecord.INDEX_NAME) || modelName.equals(LogRecord.INDEX_NAME)) {
+ map.put(IoTDBClient.TIMESTAMP, map.get("\"" + IoTDBClient.TIMESTAMP + "\""));
+ }
+ for (String key : map.keySet()) {
+ // remove double quotes
+ if (key.contains(".")) {
+ map.put(key.substring(1, key.length() - 1), map.get(key));
Review comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713751073
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/profile/IoTDBProfileThreadSnapshotQueryDAO.java
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.profile;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import org.apache.skywalking.oap.server.core.analysis.IDManager;
+import org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord;
+import org.apache.skywalking.oap.server.core.profile.ProfileThreadSnapshotRecord;
+import org.apache.skywalking.oap.server.core.query.type.BasicTrace;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.library.util.BooleanUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+public class IoTDBProfileThreadSnapshotQueryDAO implements IProfileThreadSnapshotQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<ProfileThreadSnapshotRecord> profileThreadSnapshotRecordBuilder = new ProfileThreadSnapshotRecord.Builder();
+ private final StorageHashMapBuilder<SegmentRecord> segmentRecordBuilder = new SegmentRecord.Builder();
+
+ public IoTDBProfileThreadSnapshotQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public List<BasicTrace> queryProfiledSegments(String taskId) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT)
+ .append(ProfileThreadSnapshotRecord.INDEX_NAME);
+ query = client.addQueryAsterisk(ProfileThreadSnapshotRecord.INDEX_NAME, query);
+ query.append(" where ").append(ProfileThreadSnapshotRecord.TASK_ID).append(" = \"").append(taskId).append("\"")
+ .append(" and ").append(ProfileThreadSnapshotRecord.SEQUENCE).append(" = 0")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(ProfileThreadSnapshotRecord.INDEX_NAME,
+ query.toString(), profileThreadSnapshotRecordBuilder);
+ // We can insure the size of List, so use ArrayList to improve visit speed. (Other storage plugin use LinkedList)
+ final List<String> segmentIds = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> segmentIds.add(((ProfileThreadSnapshotRecord) storageData).getSegmentId()));
+ if (segmentIds.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(SegmentRecord.INDEX_NAME);
+ query = client.addQueryAsterisk(SegmentRecord.INDEX_NAME, query);
+ query.append(" where ").append(SegmentRecord.SEGMENT_ID).append(" in (");
+ for (String segmentId : segmentIds) {
+ query.append("\"").append(segmentId).append("\"").append(", ");
+ }
+ String queryString = query.substring(0, query.length() - 2);
+ queryString = queryString + ")" + IoTDBClient.ALIGN_BY_DEVICE;
+
+ storageDataList = client.filterQuery(SegmentRecord.INDEX_NAME, queryString, segmentRecordBuilder);
+ List<SegmentRecord> segmentRecordList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> segmentRecordList.add((SegmentRecord) storageData));
+ // resort by self, because of the select query result order by time.
+ segmentRecordList.sort((SegmentRecord r1, SegmentRecord r2) -> Long.compare(r2.getStartTime(), r1.getStartTime()));
+
+ List<BasicTrace> result = new ArrayList<>(segmentRecordList.size());
+ segmentRecordList.forEach(segmentRecord -> {
+ BasicTrace basicTrace = new BasicTrace();
+ basicTrace.setSegmentId(segmentRecord.getSegmentId());
+ basicTrace.setStart(String.valueOf(segmentRecord.getStartTime()));
+ basicTrace.getEndpointNames().add(String.valueOf(IDManager.EndpointID.analysisId(segmentRecord.getEndpointId())));
Review comment:
It has been initialized in constructor and won't be null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-924691390
@LIU-WEI-git You need to
1. Make CI passed, including the new added e2e testing.
2. Remember `rebase` the commit logs, 81 commits seem crazy, even SkyWalking would do `squash and merge`, still, we hope the commit log is clear.
3. Document should be update accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-925705128
I will package a IoTDB dev docker image later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r757276305
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String GROUP_IDX = "service_group";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getMeasurements(), request.getMeasurementTypes(), request.getMeasurementValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ if (log.isDebugEnabled()) {
+ for (IoTDBInsertRequest request : requestList) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+ }
+
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getMeasurements());
+ typesList.add(request.getMeasurementTypes());
+ valuesList.add(request.getMeasurementValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ SessionDataSetWrapper wrapper = null;
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
Review comment:
`field` won't be null. What's more, if IoTDB stores a null value, `field.getDataType()` will return null and this condition will be deal with in the line 223.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] dmsolr commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
dmsolr commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755151412
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBBrowserLogQueryDAO.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.browser.source.BrowserErrorCategory;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLog;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLogs;
+import org.apache.skywalking.oap.server.core.query.type.ErrorCategory;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.library.util.StringUtil;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBBrowserLogQueryDAO implements IBrowserLogQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<BrowserErrorLogRecord> storageBuilder = new BrowserErrorLogRecord.Builder();
+
+ @Override
+ public BrowserErrorLogs queryBrowserErrorLogs(String serviceId, String serviceVersionId, String pagePathId,
+ BrowserErrorCategory category, long startSecondTB,
+ long endSecondTB, int limit, int from) throws IOException {
+ StringBuilder query = new StringBuilder();
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query.append("select * from ");
+ query = client.addModelPath(query, BrowserErrorLogRecord.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ if (StringUtil.isNotEmpty(serviceId)) {
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ }
+ query = client.addQueryIndexValue(BrowserErrorLogRecord.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where 1=1");
+ if (startSecondTB != 0 && endSecondTB != 0) {
+ query.append(" and ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startSecondTB));
+ query.append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endSecondTB));
+ }
+ if (StringUtil.isNotEmpty(serviceVersionId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.SERVICE_VERSION_ID).append(" = \"").append(serviceVersionId).append("\"");
+ }
+ if (StringUtil.isNotEmpty(pagePathId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.PAGE_PATH_ID).append(" = \"").append(pagePathId).append("\"");
+ }
+ if (Objects.nonNull(category)) {
+ query.append(" and ").append(BrowserErrorLogRecord.ERROR_CATEGORY).append(" = ").append(category.getValue());
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString();
+ queryString = queryString.replace("1=1 and ", "");
+
+ List<? super StorageData> storageDataList = client.filterQuery(BrowserErrorLogRecord.INDEX_NAME, queryString, storageBuilder);
+ List<BrowserErrorLogRecord> browserErrorLogRecordList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> browserErrorLogRecordList.add((BrowserErrorLogRecord) storageData));
+ // resort by self, because of the select query result order by time.
+ browserErrorLogRecordList.sort((BrowserErrorLogRecord b1, BrowserErrorLogRecord b2) -> Long.compare(b2.getTimestamp(), b1.getTimestamp()));
+ BrowserErrorLogs logs = new BrowserErrorLogs();
+ int limitCount = 0;
+ for (int i = 0; i < browserErrorLogRecordList.size(); i++) {
+ if (i >= from && limitCount < limit) {
+ limitCount++;
Review comment:
Why do not we start from `start` in the loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r756631251
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBAggregationQueryDAO.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.input.TopNCondition;
+import org.apache.skywalking.oap.server.core.query.type.KeyValue;
+import org.apache.skywalking.oap.server.core.query.type.SelectedRecord;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBAggregationQueryDAO implements IAggregationQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<SelectedRecord> sortMetrics(TopNCondition condition, String valueColumnName, Duration duration,
+ List<KeyValue> additionalConditions) throws IOException {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without aggregation function.
+ // https://github.com/apache/iotdb/issues/4006
+ StringBuilder query = new StringBuilder();
+ query.append(String.format("select %s from ", valueColumnName));
+ query = client.addModelPath(query, condition.getName());
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(duration.getStartTimestamp())
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(duration.getStartTimestamp());
+ if (additionalConditions != null) {
+ for (KeyValue additionalCondition : additionalConditions) {
+ query.append(" and ").append(additionalCondition.getKey()).append(" = \"")
+ .append(additionalCondition.getValue()).append("\"");
+ }
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<SelectedRecord> topEntities = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+
+ Map<String, Double> entityIdAndSumMap = new HashMap<>();
+ Map<String, Integer> entityIdAndCountMap = new HashMap<>();
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ double value = Double.parseDouble(fields.get(1).getStringValue());
+ entityIdAndSumMap.merge(entityId, value, Double::sum);
+ entityIdAndCountMap.merge(entityId, 1, Integer::sum);
+ }
+
+ entityIdAndSumMap.forEach((String entityId, Double sum) -> {
+ double count = entityIdAndCountMap.get(entityId);
+ double avg = sum / count;
+ SelectedRecord topNEntity = new SelectedRecord();
+ topNEntity.setId(entityId);
+ topNEntity.setValue(String.valueOf(avg));
Review comment:
Cast this to `long`
```suggestion
topNEntity.setValue(String.valueOf((long) avg));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r757208332
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String GROUP_IDX = "service_group";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getMeasurements(), request.getMeasurementTypes(), request.getMeasurementValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ if (log.isDebugEnabled()) {
+ for (IoTDBInsertRequest request : requestList) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+ }
+
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getMeasurements());
+ typesList.add(request.getMeasurementTypes());
+ valuesList.add(request.getMeasurementValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ SessionDataSetWrapper wrapper = null;
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
Review comment:
Specifically, this might cause the service group name to be “null” in UI topology.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r757283562
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String GROUP_IDX = "service_group";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getMeasurements(), request.getMeasurementTypes(), request.getMeasurementValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ if (log.isDebugEnabled()) {
+ for (IoTDBInsertRequest request : requestList) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+ }
+
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getMeasurements());
+ typesList.add(request.getMeasurementTypes());
+ valuesList.add(request.getMeasurementValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ SessionDataSetWrapper wrapper = null;
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
Review comment:
Please check locally. The group in topology is "null", which is wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng merged pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng merged pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-974664600
@kezhenxu94 @JaredTan95 @dmsolr Could you check this PR carefully? I think this would be the biggest PR in this iteration.
We should be careful. Although all E2E tests passed, I would recommend to verify this PR locally and check visualize on UI manually, at least this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753768143
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
It still has lz4-1.3.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753772040
##########
File path: docs/en/setup/backend/backend-storage.md
##########
@@ -280,6 +280,24 @@ storage:
All connection-related settings, including URL link, username, and password are found in `application.yml`.
Only part of the settings are listed here. Please follow [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool document for full settings.
+## IoTDB
Review comment:
Got it, I will fix it in the next commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715234331
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
Review comment:
> pkg:maven/org.apache.iotdb/iotdb-session@0.12.2
12 Critical, 2 Severe, 0 Moderate, 0 Unknown vulnerabilities have been found across 1 dependencies
@LIU-WEI-git @jixuan1989 Could you confirm the CVEs? There are 12 critical and 2 severe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715360722
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
Review comment:
I also pinged @jixuan1989 this morning, asked the same question. Wait for confirmation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-924700236
> Although I have run and tested this IoTDB storage plugin locally, I know it is not enough and it need e2e tests at least. Hope you review my work and give me some suggestions.
The e2e tests could run locally too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713729385
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -31,8 +31,7 @@
<packaging>jar</packaging>
<properties>
- <maven.compiler.source>11</maven.compiler.source>
- <maven.compiler.target>11</maven.compiler.target>
+ <iotdb-session.version>0.12.2</iotdb-session.version>
Review comment:
IoTDB client needs JDK>=1.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-925653356
The docker image is update on Sep 12, but the fuzzy query bug is fixed on [Sep 14](https://github.com/apache/iotdb/pull/3953). So some UTs cannot pass. We have to wait IoTDB version 0.12.3 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713708492
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,55 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <version>0.12.2</version>
Review comment:
Make it a property and move it to `server-bom` module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755149330
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBTableInstaller.java
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelInstaller;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+
+/**
Review comment:
ok, I will replace it with blog link in the next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r756633423
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBAggregationQueryDAO.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.input.TopNCondition;
+import org.apache.skywalking.oap.server.core.query.type.KeyValue;
+import org.apache.skywalking.oap.server.core.query.type.SelectedRecord;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBAggregationQueryDAO implements IAggregationQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<SelectedRecord> sortMetrics(TopNCondition condition, String valueColumnName, Duration duration,
+ List<KeyValue> additionalConditions) throws IOException {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without aggregation function.
+ // https://github.com/apache/iotdb/issues/4006
+ StringBuilder query = new StringBuilder();
+ query.append(String.format("select %s from ", valueColumnName));
+ query = client.addModelPath(query, condition.getName());
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(duration.getStartTimestamp())
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(duration.getStartTimestamp());
+ if (additionalConditions != null) {
+ for (KeyValue additionalCondition : additionalConditions) {
+ query.append(" and ").append(additionalCondition.getKey()).append(" = \"")
+ .append(additionalCondition.getValue()).append("\"");
+ }
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<SelectedRecord> topEntities = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+
+ Map<String, Double> entityIdAndSumMap = new HashMap<>();
+ Map<String, Integer> entityIdAndCountMap = new HashMap<>();
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ double value = Double.parseDouble(fields.get(1).getStringValue());
+ entityIdAndSumMap.merge(entityId, value, Double::sum);
+ entityIdAndCountMap.merge(entityId, 1, Integer::sum);
+ }
+
+ entityIdAndSumMap.forEach((String entityId, Double sum) -> {
+ double count = entityIdAndCountMap.get(entityId);
+ double avg = sum / count;
+ SelectedRecord topNEntity = new SelectedRecord();
+ topNEntity.setId(entityId);
+ topNEntity.setValue(String.valueOf(avg));
Review comment:
We only have `long` type value in metrics so this double should be cast to `long`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753770347
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,63 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.9.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ <version>1.6.0</version>
+ </dependency>
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753783308
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBTableInstaller.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelInstaller;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+
+public class IoTDBTableInstaller extends ModelInstaller {
Review comment:
I will deal with it these days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753783280
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBInsertRequest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+import org.apache.skywalking.oap.server.library.client.request.InsertRequest;
+import org.apache.skywalking.oap.server.library.client.request.UpdateRequest;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+
+@Getter
+@Setter
+@ToString
+@Slf4j
+public class IoTDBInsertRequest implements InsertRequest, UpdateRequest {
+ private String modelName;
+ private long time;
+ private List<String> indexes;
+ private List<String> indexValues;
+ private List<String> measurements;
+ private List<TSDataType> measurementTypes;
+ private List<Object> measurementValues;
+
+ public <T extends StorageData> IoTDBInsertRequest(String modelName, long time, T storageData,
+ StorageHashMapBuilder<T> storageBuilder) {
+ this.modelName = modelName;
+ this.time = time;
+ indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexValues = new ArrayList<>(indexes.size());
+ Map<String, Object> storageMap = storageBuilder.entity2Storage(storageData);
+
+ indexes.forEach(index -> {
+ if (index.equals(IoTDBClient.ID_IDX)) {
+ indexValues.add(storageData.id());
+ } else if (storageMap.containsKey(index)) {
+ indexValues.add(String.valueOf(storageMap.get(index)));
+ storageMap.remove(index);
+ }
+ });
+
+ // time_bucket has changed to time before calling this method, so remove it from measurements
+ storageMap.remove(IoTDBClient.TIME_BUCKET);
+ // processing illegal value
+ Iterator<Map.Entry<String, Object>> entryIterator = storageMap.entrySet().iterator();
+ while (entryIterator.hasNext()) {
+ Map.Entry<String, Object> entry = entryIterator.next();
+ if (entry.getValue() == null) {
+ entryIterator.remove();
+ }
+ if (entry.getValue() instanceof StorageDataComplexObject) {
+ storageMap.put(entry.getKey(), ((StorageDataComplexObject) entry.getValue()).toStorageData());
+ }
+ }
+
+ measurements = new ArrayList<>(storageMap.keySet());
+ Map<String, TSDataType> columnAndTypeMap = IoTDBTableMetaInfo.get(modelName).getColumnAndTypeMap();
+ measurementTypes = new ArrayList<>(measurements.size());
+ for (String measurement : measurements) {
+ measurementTypes.add(columnAndTypeMap.get(measurement));
+ }
+ measurementValues = new ArrayList<>(storageMap.values());
+
+ // processing illegal measurement
Review comment:
Ok, I will rewrite it to make it clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-974664600
@kezhenxu94 @JaredTan95 @dmsolr Could you check this PR carefully? I think this would be the biggest PR in this iteration.
We should be careful. Although all E2E tests passed, I would recommend to verify this PR locally and check visualize on UI manually, at least this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755856800
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBTopologyQueryDAO.java
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.endpoint.EndpointRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.query.type.Call;
+import org.apache.skywalking.oap.server.core.source.DetectPoint;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBTopologyQueryDAO implements ITopologyQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtServerSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationServerSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtClientSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationClientSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadEndpointRelation(long startTB, long endTB, String destEndpointId) throws IOException {
+ List<Call.CallDetail> calls = loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, false);
+ calls.addAll(loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, true));
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ List<String> serviceIds, DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ if (serviceIds.size() > 0) {
+ query.append(" and (");
+ for (int i = 0; i < serviceIds.size(); i++) {
+ query.append(sourceCName).append(" = \"").append(serviceIds.get(i))
+ .append("\" or ")
+ .append(destCName).append(" = \"").append(serviceIds.get(i)).append("\"");
+ if (i != serviceIds.size() - 1) {
+ query.append(" or ");
+ }
+ }
+ query.append(")");
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromServiceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceInstanceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String descCName,
+ String sourceServiceId, String destServiceId,
+ DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceInstanceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ((").append(sourceCName).append(" = \"").append(sourceServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(destServiceId).append("\"")
+ .append(") or (").append(sourceCName).append(" = \"").append(destServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(sourceServiceId).append(" \"))")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromInstanceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadEndpointFromSide(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ String id, boolean isSourceId) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ").append(isSourceId ? sourceCName : destCName).append(" = \"").append(id).append("\"")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ // TODO check DetectPoint
+ call.buildFromEndpointRelation(entityId, DetectPoint.SERVER);
Review comment:
For endpoint, yes. The dependency is only `server` side detection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r756624129
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String GROUP_IDX = "service_group";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getMeasurements(), request.getMeasurementTypes(), request.getMeasurementValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ if (log.isDebugEnabled()) {
+ for (IoTDBInsertRequest request : requestList) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+ }
+
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getMeasurements());
+ typesList.add(request.getMeasurementTypes());
+ valuesList.add(request.getMeasurementValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ SessionDataSetWrapper wrapper = null;
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
+ }
+ if (map.containsKey(IoTDBClient.NODE_TYPE_IDX)) {
+ String nodeType = (String) map.get(IoTDBClient.NODE_TYPE_IDX);
+ map.put(IoTDBClient.NODE_TYPE_IDX, Integer.valueOf(nodeType));
+ }
+ if (modelName.equals(BrowserErrorLogRecord.INDEX_NAME) || modelName.equals(LogRecord.INDEX_NAME)) {
+ map.put(IoTDBClient.TIMESTAMP, map.get("\"" + IoTDBClient.TIMESTAMP + "\""));
+ }
+ for (Map.Entry<String, Object> entry : map.entrySet()) {
+ // remove double quotes
+ String key = entry.getKey();
+ if (key.contains(".")) {
+ map.put(key.substring(1, key.length() - 1), entry.getValue());
+ }
+ }
+
+ storageDataList.add(storageBuilder.storage2Entity(map));
+ }
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e.getMessage() + System.lineSeparator() + "SQL Statement: " + querySQL, e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
Review comment:
Add a null check for `wrapper` in case `wrapper = sessionPool.executeQueryStatement(querySQL);` throws exception and `wrapper` is still null, the same for many other places
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBAggregationQueryDAO.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.input.TopNCondition;
+import org.apache.skywalking.oap.server.core.query.type.KeyValue;
+import org.apache.skywalking.oap.server.core.query.type.SelectedRecord;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBAggregationQueryDAO implements IAggregationQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<SelectedRecord> sortMetrics(TopNCondition condition, String valueColumnName, Duration duration,
+ List<KeyValue> additionalConditions) throws IOException {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without aggregation function.
+ // https://github.com/apache/iotdb/issues/4006
+ StringBuilder query = new StringBuilder();
+ query.append(String.format("select %s from ", valueColumnName));
+ query = client.addModelPath(query, condition.getName());
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(duration.getStartTimestamp())
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(duration.getStartTimestamp());
Review comment:
```suggestion
.append(" and ").append(IoTDBClient.TIME).append(" <= ").append(duration.getEndTimestamp());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755799051
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBTopologyQueryDAO.java
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.endpoint.EndpointRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.query.type.Call;
+import org.apache.skywalking.oap.server.core.source.DetectPoint;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBTopologyQueryDAO implements ITopologyQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtServerSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationServerSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtClientSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationClientSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadEndpointRelation(long startTB, long endTB, String destEndpointId) throws IOException {
+ List<Call.CallDetail> calls = loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, false);
+ calls.addAll(loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, true));
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ List<String> serviceIds, DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ if (serviceIds.size() > 0) {
+ query.append(" and (");
+ for (int i = 0; i < serviceIds.size(); i++) {
+ query.append(sourceCName).append(" = \"").append(serviceIds.get(i))
+ .append("\" or ")
+ .append(destCName).append(" = \"").append(serviceIds.get(i)).append("\"");
+ if (i != serviceIds.size() - 1) {
+ query.append(" or ");
+ }
+ }
+ query.append(")");
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromServiceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceInstanceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String descCName,
+ String sourceServiceId, String destServiceId,
+ DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceInstanceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ((").append(sourceCName).append(" = \"").append(sourceServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(destServiceId).append("\"")
+ .append(") or (").append(sourceCName).append(" = \"").append(destServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(sourceServiceId).append(" \"))")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromInstanceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadEndpointFromSide(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ String id, boolean isSourceId) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ").append(isSourceId ? sourceCName : destCName).append(" = \"").append(id).append("\"")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ // TODO check DetectPoint
+ call.buildFromEndpointRelation(entityId, DetectPoint.SERVER);
Review comment:
> This is client side detecting, what do you mean exactly?
No, it's not this method. In line 291 in `TopologyQueryEsDAO` and line 270 in `H2TopologyQueryDAO`, the second parameter of `buildFromEndpointRelation` is always `DetectPoint.SERVER`. But in line 305 of `TopologyQuery`, it can be `DetectPoint.SERVER` or `DetectPoint.CLIENT`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r757453777
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBInsertRequest.java
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.type.StorageDataComplexObject;
+import org.apache.skywalking.oap.server.library.client.request.InsertRequest;
+import org.apache.skywalking.oap.server.library.client.request.UpdateRequest;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBIndexes;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+
+@Getter
+@Setter
+@ToString
+@Slf4j
+public class IoTDBInsertRequest implements InsertRequest, UpdateRequest {
+ private String modelName;
+ private long time;
+ private List<String> indexes;
+ private List<String> indexValues;
+ private List<String> measurements;
+ private List<TSDataType> measurementTypes;
+ private List<Object> measurementValues;
+
+ public <T extends StorageData> IoTDBInsertRequest(String modelName, long time, T storageData,
+ StorageHashMapBuilder<T> storageBuilder) {
+ this.modelName = modelName;
+ this.time = time;
+ indexes = IoTDBTableMetaInfo.get(modelName).getIndexes();
+ indexValues = new ArrayList<>(indexes.size());
+ Map<String, Object> storageMap = storageBuilder.entity2Storage(storageData);
+
+ indexes.forEach(index -> {
+ if (index.equals(IoTDBIndexes.ID_IDX)) {
+ indexValues.add(storageData.id());
+ } else if (storageMap.containsKey(index)) {
+ // avoid `service_group` be "null" when inserting
+ if (index.equals(IoTDBIndexes.GROUP_IDX) && storageMap.get(index) == null) {
Review comment:
If other non-indexed fields is null, we won't insert it into IoTDB server. Therefore, their query results will be null, won't be "null". Please see line 81.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r744256884
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,17 +61,26 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.13.0-SNAPSHOT.jar
+iotdb-thrift-0.13.0-SNAPSHOT.jar
+istack-commons-runtime-4.0.0.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
jackson-databind-2.12.2.jar
jackson-datatype-guava-2.9.10.jar
jackson-datatype-jdk8-2.9.10.jar
jackson-module-afterburner-2.12.2.jar
jackson-module-kotlin-2.8.8.jar
+jakarta.activation-2.0.0.jar
+jakarta.xml.bind-api-3.0.0.jar
Review comment:
I build the package locally and didn't find these 2 jar as well, the same as what we see in GitHub Actions.
```diff
@@ -74,2 +72,0 @@
-jakarta.activation-2.0.0.jar
-jakarta.xml.bind-api-3.0.0.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r742502661
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -89,12 +98,16 @@ jsr305-3.0.2.jar
kafka-clients-2.4.1.jar
kotlin-reflect-1.1.1.jar
kotlin-stdlib-1.1.60.jar
+libthrift-0.14.1.jar
listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
log4j-api-2.14.1.jar
log4j-core-2.14.1.jar
log4j-over-slf4j-1.7.30.jar
log4j-slf4j-impl-2.14.1.jar
+logback-classic-1.2.3.jar
+logback-core-1.2.3.jar
logging-interceptor-3.13.1.jar
+lz4-1.3.0.jar
lz4-java-1.6.0.jar
Review comment:
OMG, these are 2 different libs. :(
- https://github.com/lz4/lz4
- https://github.com/lz4/lz4-java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-945773438
> You need to use `infra-e2e` tool when you resolve conflicts. The v1 e2e framework is being removed.
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713726686
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -31,8 +31,7 @@
<packaging>jar</packaging>
<properties>
- <maven.compiler.source>11</maven.compiler.source>
- <maven.compiler.target>11</maven.compiler.target>
+ <iotdb-session.version>0.12.2</iotdb-session.version>
Review comment:
IoTDB client doesn't require JDK 11, right? SkyWalking is widely used in JDK8 for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715279874
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,6 +61,7 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.12.2.jar
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714652137
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/README.md
##########
@@ -0,0 +1,194 @@
+Apache IoTDB - Apache Skywalking适配器
Review comment:
This kind of readme is hard up to date, so, we prefer don't keep them in official repository. If some design principles are important, such as rules of tables, and keys(indices), we should indicate them in the commenta of related codes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714651206
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/README.md
##########
@@ -0,0 +1,194 @@
+Apache IoTDB - Apache Skywalking适配器
Review comment:
This design part could land in SkyWalking website as a blog, describing the final status, such as how tables and query/index work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714639540
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBMetricsDAOTest.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.skywalking.oap.server.core.analysis.DownSampling;
+import org.apache.skywalking.oap.server.core.analysis.FunctionCategory;
+import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic;
+import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.model.ExtraQueryIndex;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBStorageConfig;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+import static org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClientTest.retrieval;
+
+public class IoTDBMetricsDAOTest {
Review comment:
Other storage tests just test their tool class. However I tests those query methods directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-925655430
Take a reference from here. https://github.com/orgs/apache/packages?repo_name=skywalking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714940627
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBLogQueryDAO.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.network.logging.v3.LogTags;
+import org.apache.skywalking.apm.util.StringUtil;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.AbstractLogRecord;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.TraceScopeCondition;
+import org.apache.skywalking.oap.server.core.query.type.ContentType;
+import org.apache.skywalking.oap.server.core.query.type.KeyValue;
+import org.apache.skywalking.oap.server.core.query.type.Log;
+import org.apache.skywalking.oap.server.core.query.type.Logs;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.ILogQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBLogQueryDAO implements ILogQueryDAO {
Review comment:
@LIU-WEI-git As you haven't override `supportQueryLogsByKeywords`, `keywordsOfContent` and `excludingKeywordsOfContent` wouldn't be supported. Please recheck from UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715285716
##########
File path: dist-material/release-docs/LICENSE
##########
@@ -327,6 +327,7 @@ The text of each license is the standard Apache 2.0 license.
Armeria 1.11.0, http://github.com/line/armeria, Apache 2.0
Brotli4j 1.6.0, https://github.com/hyperxpro/Brotli4j, Apache 2.0
micrometer 1.7.3, https://github.com/micrometer-metrics/micrometer, Apache 2.0
+ iotdb-session 0.12.2, https://github.com/apache/iotdb, Apache 2.0
Review comment:
Read the error logs, they are much more than 2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r742805724
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,17 +61,26 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.13.0-SNAPSHOT.jar
+iotdb-thrift-0.13.0-SNAPSHOT.jar
+istack-commons-runtime-4.0.0.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
jackson-databind-2.12.2.jar
jackson-datatype-guava-2.9.10.jar
jackson-datatype-jdk8-2.9.10.jar
jackson-module-afterburner-2.12.2.jar
jackson-module-kotlin-2.8.8.jar
+jakarta.activation-2.0.0.jar
+jakarta.xml.bind-api-3.0.0.jar
Review comment:
FYI @kezhenxu94 I think our maven cache may cause some issues when we use `SNAPSHOT` for draft PR.
I and @LIU-WEI-git both packaged locally, these jars exist, but CI checks on dependencies say otherwise.
```
@@ -74,2 +72,0 @@
-jakarta.activation-2.0.0.jar
-jakarta.xml.bind-api-3.0.0.jar
```
I don't think we have to fix this, as we never would accept a PR using snapshot dependency. Just find out this curious case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r742876708
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,17 +61,26 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.13.0-SNAPSHOT.jar
+iotdb-thrift-0.13.0-SNAPSHOT.jar
+istack-commons-runtime-4.0.0.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
jackson-databind-2.12.2.jar
jackson-datatype-guava-2.9.10.jar
jackson-datatype-jdk8-2.9.10.jar
jackson-module-afterburner-2.12.2.jar
jackson-module-kotlin-2.8.8.jar
+jakarta.activation-2.0.0.jar
+jakarta.xml.bind-api-3.0.0.jar
Review comment:
I think it is your local cache that cause this issue, because when maven downloads a SNAPSHOT package it will firstly check whether it exists in your local repository (typically `~/.m2/repository`), and if exists it will be used directly, but SNAPSHOT can be overrode in remote repository so you might be using an old revision locally, as I check the latest revision of this iotdb-session SNAPSHOT is `iotdb-session-0.13.0-20211102.115555-1.jar Tue Nov 02 14:07:09 UTC 2021 67297` (the same revision GitHub Action is using) so you might want to update the local SNAPSHOT and repackage to see the changes, generally adding `-U` will do the trick (`./mvnw -U ....`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r744256884
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,17 +61,26 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.13.0-SNAPSHOT.jar
+iotdb-thrift-0.13.0-SNAPSHOT.jar
+istack-commons-runtime-4.0.0.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
jackson-databind-2.12.2.jar
jackson-datatype-guava-2.9.10.jar
jackson-datatype-jdk8-2.9.10.jar
jackson-module-afterburner-2.12.2.jar
jackson-module-kotlin-2.8.8.jar
+jakarta.activation-2.0.0.jar
+jakarta.xml.bind-api-3.0.0.jar
Review comment:
I build the package and didn't find these 2 jar as well, the same as what we see in GitHub Actions.
```diff
@@ -74,2 +72,0 @@
-jakarta.activation-2.0.0.jar
-jakarta.xml.bind-api-3.0.0.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755163962
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBTableInstaller.java
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelInstaller;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+
+/**
Review comment:
Make sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755791569
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBTopologyQueryDAO.java
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.endpoint.EndpointRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.query.type.Call;
+import org.apache.skywalking.oap.server.core.source.DetectPoint;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBTopologyQueryDAO implements ITopologyQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtServerSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationServerSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtClientSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationClientSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadEndpointRelation(long startTB, long endTB, String destEndpointId) throws IOException {
+ List<Call.CallDetail> calls = loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, false);
+ calls.addAll(loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, true));
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ List<String> serviceIds, DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ if (serviceIds.size() > 0) {
+ query.append(" and (");
+ for (int i = 0; i < serviceIds.size(); i++) {
+ query.append(sourceCName).append(" = \"").append(serviceIds.get(i))
+ .append("\" or ")
+ .append(destCName).append(" = \"").append(serviceIds.get(i)).append("\"");
+ if (i != serviceIds.size() - 1) {
+ query.append(" or ");
+ }
+ }
+ query.append(")");
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromServiceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceInstanceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String descCName,
+ String sourceServiceId, String destServiceId,
+ DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceInstanceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ((").append(sourceCName).append(" = \"").append(sourceServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(destServiceId).append("\"")
+ .append(") or (").append(sourceCName).append(" = \"").append(destServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(sourceServiceId).append(" \"))")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromInstanceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadEndpointFromSide(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ String id, boolean isSourceId) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ").append(isSourceId ? sourceCName : destCName).append(" = \"").append(id).append("\"")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ // TODO check DetectPoint
+ call.buildFromEndpointRelation(entityId, DetectPoint.SERVER);
Review comment:
This is client side detecting, what do you mean exactly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755789535
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBBrowserLogQueryDAO.java
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.browser.source.BrowserErrorCategory;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLog;
+import org.apache.skywalking.oap.server.core.query.type.BrowserErrorLogs;
+import org.apache.skywalking.oap.server.core.query.type.ErrorCategory;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.library.util.StringUtil;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBBrowserLogQueryDAO implements IBrowserLogQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<BrowserErrorLogRecord> storageBuilder = new BrowserErrorLogRecord.Builder();
+
+ @Override
+ public BrowserErrorLogs queryBrowserErrorLogs(String serviceId, String serviceVersionId, String pagePathId,
+ BrowserErrorCategory category, long startSecondTB,
+ long endSecondTB, int limit, int from) throws IOException {
+ StringBuilder query = new StringBuilder();
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query.append("select * from ");
+ query = client.addModelPath(query, BrowserErrorLogRecord.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ if (StringUtil.isNotEmpty(serviceId)) {
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ }
+ query = client.addQueryIndexValue(BrowserErrorLogRecord.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where 1=1");
+ if (startSecondTB != 0 && endSecondTB != 0) {
+ query.append(" and ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startSecondTB));
+ query.append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endSecondTB));
+ }
+ if (StringUtil.isNotEmpty(serviceVersionId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.SERVICE_VERSION_ID).append(" = \"").append(serviceVersionId).append("\"");
+ }
+ if (StringUtil.isNotEmpty(pagePathId)) {
+ query.append(" and ").append(BrowserErrorLogRecord.PAGE_PATH_ID).append(" = \"").append(pagePathId).append("\"");
+ }
+ if (Objects.nonNull(category)) {
+ query.append(" and ").append(BrowserErrorLogRecord.ERROR_CATEGORY).append(" = ").append(category.getValue());
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString();
+ queryString = queryString.replace("1=1 and ", "");
Review comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713743156
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,475 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.exception.NullFieldException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String GROUP_IDX = "service_group";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) throws IOException {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ public final String getStorageGroup() {
+ return storageGroup;
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize(), config.isRpcCompression());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getTimeseriesList(), request.getTimeseriesTypes(), request.getTimeseriesValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getTimeseriesList());
+ typesList.add(request.getTimeseriesTypes());
+ valuesList.add(request.getTimeseriesValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(modelName);
+ if (!sessionPool.checkTimeseriesExists(devicePath.toString())) {
+ return storageDataList;
+ }
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
+ }
+ if (map.containsKey(IoTDBClient.NODE_TYPE_IDX)) {
+ String nodeType = (String) map.get(IoTDBClient.NODE_TYPE_IDX);
+ map.put(IoTDBClient.NODE_TYPE_IDX, Integer.valueOf(nodeType));
+ }
+ if (modelName.equals(BrowserErrorLogRecord.INDEX_NAME) || modelName.equals(LogRecord.INDEX_NAME)) {
+ map.put(IoTDBClient.TIMESTAMP, map.get("\"" + IoTDBClient.TIMESTAMP + "\""));
+ }
+ for (String key : map.keySet()) {
+ // remove double quotes
+ if (key.contains(".")) {
+ map.put(key.substring(1, key.length() - 1), map.get(key));
Review comment:
*INEFFICIENT_KEYSET_ITERATOR:* Accessing a value using a key that was retrieved from a `keySet` iterator. It is more efficient to use an iterator on the `entrySet` of the map, avoiding the extra `HashMap.get(key)` lookup.
(at-me [in a reply](https://help.sonatype.com/lift/talking-to-lift) with `help` or `ignore`)
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/profile/IoTDBProfileThreadSnapshotQueryDAO.java
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.profile;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import org.apache.skywalking.oap.server.core.analysis.IDManager;
+import org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord;
+import org.apache.skywalking.oap.server.core.profile.ProfileThreadSnapshotRecord;
+import org.apache.skywalking.oap.server.core.query.type.BasicTrace;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.library.util.BooleanUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+public class IoTDBProfileThreadSnapshotQueryDAO implements IProfileThreadSnapshotQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<ProfileThreadSnapshotRecord> profileThreadSnapshotRecordBuilder = new ProfileThreadSnapshotRecord.Builder();
+ private final StorageHashMapBuilder<SegmentRecord> segmentRecordBuilder = new SegmentRecord.Builder();
+
+ public IoTDBProfileThreadSnapshotQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public List<BasicTrace> queryProfiledSegments(String taskId) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT)
+ .append(ProfileThreadSnapshotRecord.INDEX_NAME);
+ query = client.addQueryAsterisk(ProfileThreadSnapshotRecord.INDEX_NAME, query);
+ query.append(" where ").append(ProfileThreadSnapshotRecord.TASK_ID).append(" = \"").append(taskId).append("\"")
+ .append(" and ").append(ProfileThreadSnapshotRecord.SEQUENCE).append(" = 0")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(ProfileThreadSnapshotRecord.INDEX_NAME,
+ query.toString(), profileThreadSnapshotRecordBuilder);
+ // We can insure the size of List, so use ArrayList to improve visit speed. (Other storage plugin use LinkedList)
+ final List<String> segmentIds = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> segmentIds.add(((ProfileThreadSnapshotRecord) storageData).getSegmentId()));
+ if (segmentIds.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(SegmentRecord.INDEX_NAME);
+ query = client.addQueryAsterisk(SegmentRecord.INDEX_NAME, query);
+ query.append(" where ").append(SegmentRecord.SEGMENT_ID).append(" in (");
+ for (String segmentId : segmentIds) {
+ query.append("\"").append(segmentId).append("\"").append(", ");
+ }
+ String queryString = query.substring(0, query.length() - 2);
+ queryString = queryString + ")" + IoTDBClient.ALIGN_BY_DEVICE;
+
+ storageDataList = client.filterQuery(SegmentRecord.INDEX_NAME, queryString, segmentRecordBuilder);
+ List<SegmentRecord> segmentRecordList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> segmentRecordList.add((SegmentRecord) storageData));
+ // resort by self, because of the select query result order by time.
+ segmentRecordList.sort((SegmentRecord r1, SegmentRecord r2) -> Long.compare(r2.getStartTime(), r1.getStartTime()));
+
+ List<BasicTrace> result = new ArrayList<>(segmentRecordList.size());
+ segmentRecordList.forEach(segmentRecord -> {
+ BasicTrace basicTrace = new BasicTrace();
+ basicTrace.setSegmentId(segmentRecord.getSegmentId());
+ basicTrace.setStart(String.valueOf(segmentRecord.getStartTime()));
+ basicTrace.getEndpointNames().add(String.valueOf(IDManager.EndpointID.analysisId(segmentRecord.getEndpointId())));
Review comment:
*NULL_DEREFERENCE:* object returned by `getEndpointNames(basicTrace)` could be null and is dereferenced at line 86.
(at-me [in a reply](https://help.sonatype.com/lift/talking-to-lift) with `help` or `ignore`)
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBAlarmQueryDAO.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.skywalking.oap.server.core.alarm.AlarmRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
+import org.apache.skywalking.oap.server.core.query.enumeration.Scope;
+import org.apache.skywalking.oap.server.core.query.type.AlarmMessage;
+import org.apache.skywalking.oap.server.core.query.type.Alarms;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IAlarmQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+public class IoTDBAlarmQueryDAO implements IAlarmQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<AlarmRecord> storageBuilder = new AlarmRecord.Builder();
+
+ public IoTDBAlarmQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public Alarms getAlarm(Integer scopeId, String keyword, int limit, int from, long startTB, long endTB, List<Tag> tags) throws IOException {
+ StringBuilder query = new StringBuilder();
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(AlarmRecord.INDEX_NAME);
+ query = client.addQueryAsterisk(AlarmRecord.INDEX_NAME, query);
+ query.append(" where 1=1");
+ if (Objects.nonNull(scopeId)) {
+ query.append(" and ").append(AlarmRecord.SCOPE).append(" = ").append(scopeId);
+ }
+ if (startTB != 0 && endTB != 0) {
+ query.append(" and ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB));
+ query.append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ }
+ if (!Strings.isNullOrEmpty(keyword)) {
+ query.append(" and ").append(AlarmRecord.ALARM_MESSAGE).append(" like '%").append(keyword).append("%'");
+ }
+ if (CollectionUtils.isNotEmpty(tags)) {
+ for (final Tag tag : tags) {
+ query.append(" and ").append(tag.getKey()).append(" = \"").append(tag.getValue()).append("\"");
+ }
+ }
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString().replace("1=1 and ", "");
+ queryString = queryString + IoTDBClient.ALIGN_BY_DEVICE;
+
+ Alarms alarms = new Alarms();
+ List<? super StorageData> storageDataList = client.filterQuery(AlarmRecord.INDEX_NAME, queryString, storageBuilder);
+ int limitCount = 0;
+ for (int i = 0; i < storageDataList.size(); i++) {
+ if (i >= from && limitCount < limit) {
+ limitCount++;
+ AlarmRecord alarmRecord = (AlarmRecord) storageDataList.get(i);
+ alarms.getMsgs().add(parseMessage(alarmRecord));
+ }
+ }
+ alarms.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ alarms.getMsgs().sort((AlarmMessage m1, AlarmMessage m2) -> Long.compare(m2.getStartTime(), m1.getStartTime()));
Review comment:
*NULL_DEREFERENCE:* object returned by `getMsgs(alarms)` could be null and is dereferenced at line 84.
(at-me [in a reply](https://help.sonatype.com/lift/talking-to-lift) with `help` or `ignore`)
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBEventQueryDAO.java
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import org.apache.skywalking.oap.server.core.query.PaginationUtils;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.type.event.EventQueryCondition;
+import org.apache.skywalking.oap.server.core.query.type.event.EventType;
+import org.apache.skywalking.oap.server.core.query.type.event.Events;
+import org.apache.skywalking.oap.server.core.query.type.event.Source;
+import org.apache.skywalking.oap.server.core.source.Event;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+public class IoTDBEventQueryDAO implements IEventQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<Event> storageBuilder = new Event.Builder();
+
+ public IoTDBEventQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public Events queryEvents(EventQueryCondition condition) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(condition, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = 0; i < storageDataList.size(); i++) {
+ if (i >= page.getFrom() && limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ org.apache.skywalking.oap.server.core.query.type.event.Event resultEvent = parseEvent(event);
+ events.getEvents().add(resultEvent);
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
+ (org.apache.skywalking.oap.server.core.query.type.event.Event e1,
+ org.apache.skywalking.oap.server.core.query.type.event.Event e2)
+ -> Long.compare(e2.getStartTime(), e1.getStartTime()));
+ } else {
+ events.getEvents().sort(
+ Comparator.comparingLong(org.apache.skywalking.oap.server.core.query.type.event.Event::getStartTime));
+ }
+ return events;
+ }
+
+ @Override
+ public Events queryEvents(List<EventQueryCondition> conditionList) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(conditionList, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ EventQueryCondition condition = conditionList.get(0);
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = 0; i < storageDataList.size(); i++) {
+ if (i >= page.getFrom() && limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ org.apache.skywalking.oap.server.core.query.type.event.Event resultEvent = parseEvent(event);
+ events.getEvents().add(resultEvent);
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
Review comment:
*NULL_DEREFERENCE:* object returned by `getEvents(events)` could be null and is dereferenced at line 110.
(at-me [in a reply](https://help.sonatype.com/lift/talking-to-lift) with `help` or `ignore`)
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBEventQueryDAO.java
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import org.apache.skywalking.oap.server.core.query.PaginationUtils;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.type.event.EventQueryCondition;
+import org.apache.skywalking.oap.server.core.query.type.event.EventType;
+import org.apache.skywalking.oap.server.core.query.type.event.Events;
+import org.apache.skywalking.oap.server.core.query.type.event.Source;
+import org.apache.skywalking.oap.server.core.source.Event;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+public class IoTDBEventQueryDAO implements IEventQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<Event> storageBuilder = new Event.Builder();
+
+ public IoTDBEventQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public Events queryEvents(EventQueryCondition condition) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(condition, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = 0; i < storageDataList.size(); i++) {
+ if (i >= page.getFrom() && limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ org.apache.skywalking.oap.server.core.query.type.event.Event resultEvent = parseEvent(event);
+ events.getEvents().add(resultEvent);
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
+ (org.apache.skywalking.oap.server.core.query.type.event.Event e1,
+ org.apache.skywalking.oap.server.core.query.type.event.Event e2)
+ -> Long.compare(e2.getStartTime(), e1.getStartTime()));
+ } else {
+ events.getEvents().sort(
Review comment:
*NULL_DEREFERENCE:* object returned by `getEvents(events)` could be null and is dereferenced at line 77.
(at-me [in a reply](https://help.sonatype.com/lift/talking-to-lift) with `help` or `ignore`)
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBMetadataQueryDAO.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import com.google.gson.Gson;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import java.util.HashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.util.StringUtil;
+import org.apache.skywalking.oap.server.core.analysis.NodeType;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.endpoint.EndpointTraffic;
+import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic;
+import org.apache.skywalking.oap.server.core.analysis.manual.service.ServiceTraffic;
+import org.apache.skywalking.oap.server.core.query.enumeration.Language;
+import org.apache.skywalking.oap.server.core.query.type.Attribute;
+import org.apache.skywalking.oap.server.core.query.type.Database;
+import org.apache.skywalking.oap.server.core.query.type.Endpoint;
+import org.apache.skywalking.oap.server.core.query.type.Service;
+import org.apache.skywalking.oap.server.core.query.type.ServiceInstance;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IMetadataQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+@Slf4j
+public class IoTDBMetadataQueryDAO implements IMetadataQueryDAO {
+ private static final Gson GSON = new Gson();
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<ServiceTraffic> serviceBuilder = new ServiceTraffic.Builder();
+ private final StorageHashMapBuilder<EndpointTraffic> endpointBuilder = new EndpointTraffic.Builder();
+ private final StorageHashMapBuilder<InstanceTraffic> instanceBuilder = new InstanceTraffic.Builder();
+
+ public IoTDBMetadataQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public List<Service> getAllServices(String group) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(ServiceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.NODE_TYPE_IDX, String.valueOf(NodeType.Normal.value()));
+ if (StringUtil.isNotEmpty(group)) {
+ indexAndValueMap.put(IoTDBClient.GROUP_IDX, group);
+ }
+ query = client.addQueryIndexValue(ServiceTraffic.INDEX_NAME, query, indexAndValueMap);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(ServiceTraffic.INDEX_NAME, query.toString(), serviceBuilder);
+ List<Service> serviceList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> serviceList.add(buildService((ServiceTraffic) storageData)));
+ return serviceList;
+ }
+
+ @Override
+ public List<Service> getAllBrowserServices() throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(ServiceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.NODE_TYPE_IDX, String.valueOf(NodeType.Browser.value()));
+ query = client.addQueryIndexValue(ServiceTraffic.INDEX_NAME, query, indexAndValueMap);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(ServiceTraffic.INDEX_NAME, query.toString(), serviceBuilder);
+ List<Service> serviceList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> serviceList.add(buildService((ServiceTraffic) storageData)));
+ return serviceList;
+ }
+
+ @Override
+ public List<Database> getAllDatabases() throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(ServiceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.NODE_TYPE_IDX, String.valueOf(NodeType.Database.value()));
+ query = client.addQueryIndexValue(ServiceTraffic.INDEX_NAME, query, indexAndValueMap);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(ServiceTraffic.INDEX_NAME, query.toString(), serviceBuilder);
+ List<Database> databaseList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> {
+ ServiceTraffic serviceTraffic = (ServiceTraffic) storageData;
+ Database database = new Database();
+ database.setId(serviceTraffic.id());
+ database.setName(serviceTraffic.getName());
+ databaseList.add(database);
+ });
+ return databaseList;
+ }
+
+ @Override
+ public List<Service> searchServices(String keyword) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(ServiceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.NODE_TYPE_IDX, String.valueOf(NodeType.Normal.value()));
+ query = client.addQueryIndexValue(ServiceTraffic.INDEX_NAME, query, indexAndValueMap);
+ if (!Strings.isNullOrEmpty(keyword)) {
+ query.append(" where ").append(ServiceTraffic.NAME).append(" like '%").append(keyword).append("%'");
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+ List<? super StorageData> storageDataList = client.filterQuery(ServiceTraffic.INDEX_NAME, query.toString(), serviceBuilder);
+ List<Service> serviceList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> serviceList.add(buildService((ServiceTraffic) storageData)));
+ return serviceList;
+ }
+
+ @Override
+ public Service searchService(String serviceCode) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(ServiceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.NODE_TYPE_IDX, String.valueOf(NodeType.Normal.value()));
+ query = client.addQueryIndexValue(ServiceTraffic.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where ").append(ServiceTraffic.NAME).append(" = \"").append(serviceCode).append("\"")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+ List<? super StorageData> storageDataList = client.filterQuery(ServiceTraffic.INDEX_NAME, query.toString(), serviceBuilder);
+ return buildService((ServiceTraffic) storageDataList.get(0));
+ }
+
+ @Override
+ public List<Endpoint> searchEndpoint(String keyword, String serviceId, int limit) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(EndpointTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ query = client.addQueryIndexValue(EndpointTraffic.INDEX_NAME, query, indexAndValueMap);
+ if (!Strings.isNullOrEmpty(keyword)) {
+ query.append(" where ").append(EndpointTraffic.NAME).append(" like '%").append(keyword).append("%'");
+ }
+ query.append(" limit ").append(limit).append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(EndpointTraffic.INDEX_NAME, query.toString(), endpointBuilder);
+ List<Endpoint> endpointList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> {
+ EndpointTraffic endpointTraffic = (EndpointTraffic) storageData;
+ Endpoint endpoint = new Endpoint();
+ endpoint.setId(endpointTraffic.id());
+ endpoint.setName(endpointTraffic.getName());
+ endpointList.add(endpoint);
+ });
+ return endpointList;
+ }
+
+ @Override
+ public List<ServiceInstance> getServiceInstances(long startTimestamp, long endTimestamp, String serviceId) throws IOException {
+ final long minuteTimeBucket = TimeBucket.getMinuteTimeBucket(startTimestamp);
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(InstanceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ query = client.addQueryIndexValue(InstanceTraffic.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where ").append(InstanceTraffic.LAST_PING_TIME_BUCKET).append(" >= ").append(minuteTimeBucket)
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(InstanceTraffic.INDEX_NAME, query.toString(), instanceBuilder);
+ List<ServiceInstance> serviceInstanceList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> {
+ InstanceTraffic instanceTraffic = (InstanceTraffic) storageData;
+ ServiceInstance serviceInstance = new ServiceInstance();
+ serviceInstance.setId(instanceTraffic.id());
+ serviceInstance.setName(instanceTraffic.getName());
+ serviceInstance.setInstanceUUID(serviceInstance.getId());
+
+ JsonObject properties = instanceTraffic.getProperties();
+ if (!properties.isJsonNull()) {
+ for (Map.Entry<String, JsonElement> property : properties.entrySet()) {
+ String key = property.getKey();
+ String value = property.getValue().getAsString();
+ if (key.equals(InstanceTraffic.PropertyUtil.LANGUAGE)) {
+ serviceInstance.setLanguage(Language.value(value));
+ } else {
+ serviceInstance.getAttributes().add(new Attribute(key, value));
Review comment:
*NULL_DEREFERENCE:* object returned by `getAttributes(serviceInstance)` could be null and is dereferenced at line 197.
(at-me [in a reply](https://help.sonatype.com/lift/talking-to-lift) with `help` or `ignore`)
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
Review comment:
*Critical OSS Vulnerability:*
### pkg:maven/org.apache.iotdb/iotdb-session@0.12.2
12 Critical, 2 Severe, 0 Moderate, 0 Unknown vulnerabilities have been found across 1 dependencies
<details>
<summary><b>Components</b></summary><br/>
<ul>
<details>
<summary><b>pkg:maven/org.apache.tomcat.embed/tomcat-embed-core@8.5.46</b></summary>
<ul>
<details>
<summary><b>CRITICAL Vulnerabilities (12)</b></summary><br/>
<ul>
<details>
<summary>CVE-2020-1938</summary>
> #### [CVE-2020-1938] When using the Apache JServ Protocol (AJP), care must be taken when trusting inc...
> When using the Apache JServ Protocol (AJP), care must be taken when trusting incoming connections to Apache Tomcat. Tomcat treats AJP connections as having higher trust than, for example, a similar HTTP connection. If such connections are available to an attacker, they can be exploited in ways that may be surprising. In Apache Tomcat 9.0.0.M1 to 9.0.0.30, 8.5.0 to 8.5.50 and 7.0.0 to 7.0.99, Tomcat shipped with an AJP Connector enabled by default that listened on all configured IP addresses. It was expected (and recommended in the security guide) that this Connector would be disabled if not required. This vulnerability report identified a mechanism that allowed: - returning arbitrary files from anywhere in the web application - processing any file in the web application as a JSP Further, if the web application allowed file upload and stored those files within the web application (or the attacker was able to control the content of the web application by some other means) then thi
s, along with the ability to process a file as a JSP, made remote code execution possible. It is important to note that mitigation is only required if an AJP port is accessible to untrusted users. Users wishing to take a defence-in-depth approach and block the vector that permits returning arbitrary files and execution as JSP may upgrade to Apache Tomcat 9.0.31, 8.5.51 or 7.0.100 or later. A number of changes were made to the default AJP Connector configuration in 9.0.31 to harden the default configuration. It is likely that users upgrading to 9.0.31, 8.5.51 or 7.0.100 or later will need to make small changes to their configurations.
>
> **CVSS Score:** 9.8
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2021-30640</summary>
> #### [CVE-2021-30640] A vulnerability in the JNDI Realm of Apache Tomcat allows an attacker to authent...
> A vulnerability in the JNDI Realm of Apache Tomcat allows an attacker to authenticate using variations of a valid user name and/or to bypass some of the protection provided by the LockOut Realm. This issue affects Apache Tomcat 10.0.0-M1 to 10.0.5; 9.0.0.M1 to 9.0.45; 8.5.0 to 8.5.65.
>
> **CVSS Score:** 8.2
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:H/A:N
</details>
<details>
<summary>CVE-2020-13935</summary>
> #### [CVE-2020-13935] The payload length in a WebSocket frame was not correctly validated in Apache To...
> The payload length in a WebSocket frame was not correctly validated in Apache Tomcat 10.0.0-M1 to 10.0.0-M6, 9.0.0.M1 to 9.0.36, 8.5.0 to 8.5.56 and 7.0.27 to 7.0.104. Invalid payload lengths could trigger an infinite loop. Multiple requests with invalid payload lengths could lead to a denial of service.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2019-17563</summary>
> #### [CVE-2019-17563] When using FORM authentication with Apache Tomcat 9.0.0.M1 to 9.0.29, 8.5.0 to 8...
> When using FORM authentication with Apache Tomcat 9.0.0.M1 to 9.0.29, 8.5.0 to 8.5.49 and 7.0.0 to 7.0.98 there was a narrow window where an attacker could perform a session fixation attack. The window was considered too narrow for an exploit to be practical but, erring on the side of caution, this issue has been treated as a security vulnerability.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:H/PR:N/UI:R/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2020-13934</summary>
> #### [CVE-2020-13934] An h2c direct connection to Apache Tomcat 10.0.0-M1 to 10.0.0-M6, 9.0.0.M5 to 9....
> An h2c direct connection to Apache Tomcat 10.0.0-M1 to 10.0.0-M6, 9.0.0.M5 to 9.0.36 and 8.5.1 to 8.5.56 did not release the HTTP/1.1 processor after the upgrade to HTTP/2. If a sufficient number of such requests were made, an OutOfMemoryException could occur leading to a denial of service.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2020-17527</summary>
> #### [CVE-2020-17527] While investigating bug 64830 it was discovered that Apache Tomcat 10.0.0-M1 to ...
> While investigating bug 64830 it was discovered that Apache Tomcat 10.0.0-M1 to 10.0.0-M9, 9.0.0-M1 to 9.0.39 and 8.5.0 to 8.5.59 could re-use an HTTP request header value from the previous stream received on an HTTP/2 connection for the request associated with the subsequent stream. While this would most likely lead to an error and the closure of the HTTP/2 connection, it is possible that information could leak between requests.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2020-11996</summary>
> #### [CVE-2020-11996] A specially crafted sequence of HTTP/2 requests sent to Apache Tomcat 10.0.0-M1 ...
> A specially crafted sequence of HTTP/2 requests sent to Apache Tomcat 10.0.0-M1 to 10.0.0-M5, 9.0.0.M1 to 9.0.35 and 8.5.0 to 8.5.55 could trigger high CPU usage for several seconds. If a sufficient number of such requests were made on concurrent HTTP/2 connections, the server could become unresponsive.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
</details>
<details>
<summary>CVE-2021-25122</summary>
> #### [CVE-2021-25122] When responding to new h2c connection requests, Apache Tomcat versions 10.0.0-M1...
> When responding to new h2c connection requests, Apache Tomcat versions 10.0.0-M1 to 10.0.0, 9.0.0.M1 to 9.0.41 and 8.5.0 to 8.5.61 could duplicate request headers and a limited amount of request body from one request to another meaning user A and user B could both see the results of user A's request.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2021-24122</summary>
> #### [CVE-2021-24122] When serving resources from a network location using the NTFS file system, Apach...
> When serving resources from a network location using the NTFS file system, Apache Tomcat versions 10.0.0-M1 to 10.0.0-M9, 9.0.0.M1 to 9.0.39, 8.5.0 to 8.5.59 and 7.0.0 to 7.0.106 were susceptible to JSP source code disclosure in some configurations. The root cause was the unexpected behaviour of the JRE API File.getCanonicalPath() which in turn was caused by the inconsistent behaviour of the Windows API (FindFirstFileW) in some circumstances.
>
> **CVSS Score:** 7.5
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N
</details>
<details>
<summary>CVE-2020-9484</summary>
> #### [CVE-2020-9484] When using Apache Tomcat versions 10.0.0-M1 to 10.0.0-M4, 9.0.0.M1 to 9.0.34, 8....
> When using Apache Tomcat versions 10.0.0-M1 to 10.0.0-M4, 9.0.0.M1 to 9.0.34, 8.5.0 to 8.5.54 and 7.0.0 to 7.0.103 if a) an attacker is able to control the contents and name of a file on the server; and b) the server is configured to use the PersistenceManager with a FileStore; and c) the PersistenceManager is configured with sessionAttributeValueClassNameFilter="null" (the default unless a SecurityManager is used) or a sufficiently lax filter to allow the attacker provided object to be deserialized; and d) the attacker knows the relative file path from the storage location used by FileStore to the file the attacker has control over; then, using a specifically crafted request, the attacker will be able to trigger remote code execution via deserialization of the file under their control. Note that all of conditions a) to d) must be true for the attack to succeed.
>
> **CVSS Score:** 7
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2021-25329</summary>
> #### [CVE-2021-25329] The fix for CVE-2020-9484 was incomplete. When using Apache Tomcat 10.0.0-M1 to ...
> The fix for CVE-2020-9484 was incomplete. When using Apache Tomcat 10.0.0-M1 to 10.0.0, 9.0.0.M1 to 9.0.41, 8.5.0 to 8.5.61 or 7.0.0. to 7.0.107 with a configuration edge case that was highly unlikely to be used, the Tomcat instance was still vulnerable to CVE-2020-9494. Note that both the previously published prerequisites for CVE-2020-9484 and the previously published mitigations for CVE-2020-9484 also apply to this issue.
>
> **CVSS Score:** 7
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
<details>
<summary>CVE-2019-12418</summary>
> #### [CVE-2019-12418] When Apache Tomcat 9.0.0.M1 to 9.0.28, 8.5.0 to 8.5.47, 7.0.0 and 7.0.97 is conf...
> When Apache Tomcat 9.0.0.M1 to 9.0.28, 8.5.0 to 8.5.47, 7.0.0 and 7.0.97 is configured with the JMX Remote Lifecycle Listener, a local attacker without access to the Tomcat process or configuration files is able to manipulate the RMI registry to perform a man-in-the-middle attack to capture user names and passwords used to access the JMX interface. The attacker can then use these credentials to access the JMX interface and gain complete control over the Tomcat instance.
>
> **CVSS Score:** 7
>
> **CVSS Vector:** CVSS:3.0/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:H
</details>
</ul>
</details>
<details>
<summary><b>SEVERE Vulnerabilities (2)</b></summary><br/>
<ul>
<details>
<summary>CVE-2021-33037</summary>
> #### [CVE-2021-33037] Apache Tomcat 10.0.0-M1 to 10.0.6, 9.0.0.M1 to 9.0.46 and 8.5.0 to 8.5.66 did no...
> Apache Tomcat 10.0.0-M1 to 10.0.6, 9.0.0.M1 to 9.0.46 and 8.5.0 to 8.5.66 did not correctly parse the HTTP transfer-encoding request header in some circumstances leading to the possibility to request smuggling when used with a reverse proxy. Specifically: - Tomcat incorrectly ignored the transfer encoding header if the client declared it would only accept an HTTP/1.0 response; - Tomcat honoured the identify encoding; and - Tomcat did not ensure that, if present, the chunked encoding was the final encoding.
>
> **CVSS Score:** 5.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:L/A:N
</details>
<details>
<summary>CVE-2020-13943</summary>
> #### [CVE-2020-13943] If an HTTP/2 client connecting to Apache Tomcat 10.0.0-M1 to 10.0.0-M7, 9.0.0.M1...
> If an HTTP/2 client connecting to Apache Tomcat 10.0.0-M1 to 10.0.0-M7, 9.0.0.M1 to 9.0.37 or 8.5.0 to 8.5.57 exceeded the agreed maximum number of concurrent streams for a connection (in violation of the HTTP/2 protocol), it was possible that a subsequent request made on that connection could contain HTTP headers - including HTTP/2 pseudo headers - from a previous request rather than the intended headers. This could lead to users seeing responses for unexpected resources.
>
> **CVSS Score:** 4.3
>
> **CVSS Vector:** CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N
</details>
</ul>
</details>
</ul>
</details>
</ul>
</details>
(at-me [in a reply](https://help.sonatype.com/lift/talking-to-lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-979907932
> Waiting to TopN verification. @LIU-WEI-git Please confirm when you are done.
I have confirmed it. It works well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r757287447
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String GROUP_IDX = "service_group";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getMeasurements(), request.getMeasurementTypes(), request.getMeasurementValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ if (log.isDebugEnabled()) {
+ for (IoTDBInsertRequest request : requestList) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+ }
+
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getMeasurements());
+ typesList.add(request.getMeasurementTypes());
+ valuesList.add(request.getMeasurementValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ SessionDataSetWrapper wrapper = null;
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
Review comment:
When inserting data, the `service_group` will be "null" if user doesn't set since `service_group` is one of the index and its null value is casted to "null". I have located this bug and will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-927140561
We have started 8.8.0 release, please sync to master carefully, version in pom.xml changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753774647
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBTableInstaller.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelInstaller;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+
+public class IoTDBTableInstaller extends ModelInstaller {
Review comment:
Could you consider to add a long comment here about what are typically tables look like? Such as including 5 types
1. Metadata, use `service_traffic` for example.
2. Metrics, use `service_cpm`
3. Trace segment
4. Log
5. Profiling snapshots
With all these, when we face bugs one day, or with IoTDB improvement, we will know the expected table structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755184801
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBTopologyQueryDAO.java
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.endpoint.EndpointRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.instance.ServiceInstanceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationClientSideMetrics;
+import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationServerSideMetrics;
+import org.apache.skywalking.oap.server.core.query.type.Call;
+import org.apache.skywalking.oap.server.core.source.DetectPoint;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+@Slf4j
+@RequiredArgsConstructor
+public class IoTDBTopologyQueryDAO implements ITopologyQueryDAO {
+ private final IoTDBClient client;
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB,
+ List<String> serviceIds) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ serviceIds, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationsDetectedAtServerSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationServerSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadServiceRelationDetectedAtClientSide(long startTB, long endTB) throws IOException {
+ return loadServiceCalls(
+ ServiceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceRelationClientSideMetrics.DEST_SERVICE_ID,
+ new ArrayList<>(0), DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtServerSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationServerSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationServerSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.SERVER);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadInstanceRelationDetectedAtClientSide(String clientServiceId,
+ String serverServiceId,
+ long startTB, long endTB) throws IOException {
+ return loadServiceInstanceCalls(
+ ServiceInstanceRelationClientSideMetrics.INDEX_NAME, startTB, endTB,
+ ServiceInstanceRelationClientSideMetrics.SOURCE_SERVICE_ID,
+ ServiceInstanceRelationClientSideMetrics.DEST_SERVICE_ID,
+ clientServiceId, serverServiceId, DetectPoint.CLIENT);
+ }
+
+ @Override
+ public List<Call.CallDetail> loadEndpointRelation(long startTB, long endTB, String destEndpointId) throws IOException {
+ List<Call.CallDetail> calls = loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, false);
+ calls.addAll(loadEndpointFromSide(
+ EndpointRelationServerSideMetrics.INDEX_NAME, startTB, endTB,
+ EndpointRelationServerSideMetrics.SOURCE_ENDPOINT,
+ EndpointRelationServerSideMetrics.DEST_ENDPOINT,
+ destEndpointId, true));
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ List<String> serviceIds, DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ if (serviceIds.size() > 0) {
+ query.append(" and (");
+ for (int i = 0; i < serviceIds.size(); i++) {
+ query.append(sourceCName).append(" = \"").append(serviceIds.get(i))
+ .append("\" or ")
+ .append(destCName).append(" = \"").append(serviceIds.get(i)).append("\"");
+ if (i != serviceIds.size() - 1) {
+ query.append(" or ");
+ }
+ }
+ query.append(")");
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromServiceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadServiceInstanceCalls(String tableName, long startTB, long endTB,
+ String sourceCName, String descCName,
+ String sourceServiceId, String destServiceId,
+ DetectPoint detectPoint) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select ").append(ServiceInstanceRelationServerSideMetrics.COMPONENT_ID).append(" from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ((").append(sourceCName).append(" = \"").append(sourceServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(destServiceId).append("\"")
+ .append(") or (").append(sourceCName).append(" = \"").append(destServiceId).append("\"")
+ .append(" and ").append(descCName).append(" = \"").append(sourceServiceId).append(" \"))")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ final int componentId = fields.get(1).getIntV();
+ call.buildFromInstanceRelation(entityId, componentId, detectPoint);
+ calls.add(call);
+ }
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ throw new IOException(e);
+ } finally {
+ sessionPool.closeResultSet(wrapper);
+ }
+ return calls;
+ }
+
+ private List<Call.CallDetail> loadEndpointFromSide(String tableName, long startTB, long endTB,
+ String sourceCName, String destCName,
+ String id, boolean isSourceId) throws IOException {
+ // This method don't use "group by" like other storage plugin.
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ");
+ query = client.addModelPath(query, tableName);
+ query = client.addQueryAsterisk(tableName, query);
+ query.append(" where ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB))
+ .append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ query.append(" and ").append(isSourceId ? sourceCName : destCName).append(" = \"").append(id).append("\"")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ SessionPool sessionPool = client.getSessionPool();
+ SessionDataSetWrapper wrapper = null;
+ List<Call.CallDetail> calls = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(query.toString());
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", query, wrapper.getColumnNames());
+ }
+ while (wrapper.hasNext()) {
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ Call.CallDetail call = new Call.CallDetail();
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ String entityId = client.layerName2IndexValue(layerNames[2]);
+ // TODO check DetectPoint
+ call.buildFromEndpointRelation(entityId, DetectPoint.SERVER);
Review comment:
I find the second parameter of `call.buildFromEndpointRelation` is always `DetectPoint.SERVER` in H2TopologyQueryDAO and TopologyQueryEsDAO, but not in TopologyQuery. Which one is right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r755147276
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBTableInstaller.java
##########
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelInstaller;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+
+/**
Review comment:
As you have moved this to the official website, I'd just paste the link here instead of pasting the entire content, the markdown won't be rendered correctly via JavaDoc engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r753770757
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -44,6 +44,16 @@
<dependency>
<groupId>org.apache.iotdb</groupId>
<artifactId>iotdb-session</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>net.jpountz.lz4</groupId>
+ <artifactId>lz4</artifactId>
+ </exclusion>
Review comment:
@jixuan1989 We manually replaced this with `1.6.0` to avoid 2 versions existing together in the final binary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r757283963
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/IoTDBClient.java
##########
@@ -0,0 +1,352 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.session.pool.SessionDataSetWrapper;
+import org.apache.iotdb.session.pool.SessionPool;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.log.LogRecord;
+import org.apache.skywalking.oap.server.core.browser.manual.errorlog.BrowserErrorLogRecord;
+import org.apache.skywalking.oap.server.core.management.ui.template.UITemplate;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.healthcheck.DelegatedHealthChecker;
+import org.apache.skywalking.oap.server.library.client.healthcheck.HealthCheckable;
+import org.apache.skywalking.oap.server.library.util.HealthChecker;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.base.IoTDBInsertRequest;
+
+@Slf4j
+public class IoTDBClient implements Client, HealthCheckable {
+ private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();
+ private final IoTDBStorageConfig config;
+
+ private SessionPool sessionPool;
+ private final String storageGroup;
+
+ public static final String DOT = ".";
+ public static final String ALIGN_BY_DEVICE = " align by device";
+
+ public static final String TIME_BUCKET = "time_bucket";
+ public static final String TIME = "Time";
+ public static final String TIMESTAMP = "timestamp";
+
+ public static final String ID_IDX = "id";
+ public static final String ENTITY_ID_IDX = "entity_id";
+ public static final String NODE_TYPE_IDX = "node_type";
+ public static final String SERVICE_ID_IDX = "service_id";
+ public static final String GROUP_IDX = "service_group";
+ public static final String TRACE_ID_IDX = "trace_id";
+
+ public IoTDBClient(IoTDBStorageConfig config) {
+ this.config = config;
+ storageGroup = config.getStorageGroup();
+ }
+
+ @Override
+ public void connect() throws IoTDBConnectionException, StatementExecutionException {
+ try {
+ sessionPool = new SessionPool(config.getHost(), config.getRpcPort(), config.getUsername(),
+ config.getPassword(), config.getSessionPoolSize());
+ sessionPool.setStorageGroup(storageGroup);
+
+ healthChecker.health();
+ } catch (StatementExecutionException e) {
+ if (e.getStatusCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ healthChecker.unHealth(e);
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public void shutdown() {
+ sessionPool.close();
+ this.healthChecker.health();
+ }
+
+ @Override
+ public void registerChecker(HealthChecker healthChecker) {
+ this.healthChecker.register(healthChecker);
+ }
+
+ public SessionPool getSessionPool() {
+ return sessionPool;
+ }
+
+ public IoTDBStorageConfig getConfig() {
+ return config;
+ }
+
+ /**
+ * Write data to IoTDB
+ *
+ * @param request an IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(IoTDBInsertRequest request) throws IOException {
+ if (log.isDebugEnabled()) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ try {
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ sessionPool.insertRecord(devicePath.toString(), request.getTime(),
+ request.getMeasurements(), request.getMeasurementTypes(), request.getMeasurementValues());
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Write a list of data into IoTDB
+ *
+ * @param requestList a list of IoTDBInsertRequest
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public void write(List<IoTDBInsertRequest> requestList) throws IOException {
+ if (log.isDebugEnabled()) {
+ for (IoTDBInsertRequest request : requestList) {
+ log.debug("Writing data to IoTDB: {}", request);
+ }
+ }
+
+ List<String> devicePathList = new ArrayList<>();
+ List<Long> timeList = new ArrayList<>();
+ List<List<String>> timeseriesListList = new ArrayList<>();
+ List<List<TSDataType>> typesList = new ArrayList<>();
+ List<List<Object>> valuesList = new ArrayList<>();
+
+ requestList.forEach(request -> {
+ StringBuilder devicePath = new StringBuilder();
+ devicePath.append(storageGroup).append(IoTDBClient.DOT).append(request.getModelName());
+ // make an index value as a layer name of the storage path
+ if (!request.getIndexes().isEmpty()) {
+ request.getIndexValues().forEach(value -> devicePath.append(IoTDBClient.DOT)
+ .append(indexValue2LayerName(value)));
+ }
+ devicePathList.add(devicePath.toString());
+ timeList.add(request.getTime());
+ timeseriesListList.add(request.getMeasurements());
+ typesList.add(request.getMeasurementTypes());
+ valuesList.add(request.getMeasurementValues());
+ });
+
+ try {
+ sessionPool.insertRecords(devicePathList, timeList, timeseriesListList, typesList, valuesList);
+ healthChecker.health();
+ } catch (IoTDBConnectionException | StatementExecutionException e) {
+ healthChecker.unHealth(e);
+ throw new IOException(e);
+ }
+ }
+
+ /**
+ * Normal filter query for a list of data. querySQL must contain "align by device"
+ *
+ * @param modelName model name
+ * @param querySQL the SQL for query which must contain "align by device"
+ * @param storageBuilder storage builder for transforming storage result map to entity
+ * @return a list of result data
+ * @throws IOException IoTDBConnectionException or StatementExecutionException
+ */
+ public List<? super StorageData> filterQuery(String modelName, String querySQL,
+ StorageHashMapBuilder<? extends StorageData> storageBuilder)
+ throws IOException {
+ if (!querySQL.contains("align by device")) {
+ throw new IOException("querySQL must contain \"align by device\"");
+ }
+ SessionDataSetWrapper wrapper = null;
+ List<? super StorageData> storageDataList = new ArrayList<>();
+ try {
+ wrapper = sessionPool.executeQueryStatement(querySQL);
+ if (log.isDebugEnabled()) {
+ log.debug("SQL: {}, columnNames: {}", querySQL, wrapper.getColumnNames());
+ }
+
+ List<String> columnNames = wrapper.getColumnNames();
+ IoTDBTableMetaInfo tableMetaInfo = IoTDBTableMetaInfo.get(modelName);
+ List<String> indexes = tableMetaInfo.getIndexes();
+ while (wrapper.hasNext()) {
+ Map<String, Object> map = new ConcurrentHashMap<>();
+ RowRecord rowRecord = wrapper.next();
+ List<Field> fields = rowRecord.getFields();
+ // transform timestamp to time_bucket
+ if (!UITemplate.INDEX_NAME.equals(modelName)) {
+ map.put(IoTDBClient.TIME_BUCKET, TimeBucket.getTimeBucket(rowRecord.getTimestamp(),
+ tableMetaInfo.getModel().getDownsampling()));
+ }
+ // field.get(0) -> Device, transform layerName to indexValue
+ String[] layerNames = fields.get(0).getStringValue().split("\\" + IoTDBClient.DOT + "\"");
+ for (int i = 0; i < indexes.size(); i++) {
+ map.put(indexes.get(i), layerName2IndexValue(layerNames[i + 1]));
+ }
+ for (int i = 0; i < columnNames.size() - 2; i++) {
+ String columnName = columnNames.get(i + 2);
+ Field field = fields.get(i + 1);
+ if (field.getDataType() == null) {
+ continue;
+ }
+ if (field.getDataType().equals(TSDataType.TEXT)) {
+ map.put(columnName, field.getStringValue());
+ } else {
+ map.put(columnName, field.getObjectValue(field.getDataType()));
+ }
Review comment:
I didn't dig deeper to see what's the problem but "null" string exists indeed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-924727826
E2E passed, but you face some UT failure, please fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#issuecomment-925654932
Do you have GitHub host dev image? It is fine to use it for testing. We just need to indicate release version on doc, and update test image version later.
SkyWalking is using this mode, because waiting for another version could be long and face new bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715286320
##########
File path: dist-material/release-docs/LICENSE
##########
@@ -327,6 +327,7 @@ The text of each license is the standard Apache 2.0 license.
Armeria 1.11.0, http://github.com/line/armeria, Apache 2.0
Brotli4j 1.6.0, https://github.com/hyperxpro/Brotli4j, Apache 2.0
micrometer 1.7.3, https://github.com/micrometer-metrics/micrometer, Apache 2.0
+ iotdb-session 0.12.2, https://github.com/apache/iotdb, Apache 2.0
Review comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715230475
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,6 +61,7 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.12.2.jar
Review comment:
How does IoTDB client implement the network requests? I can see there is no more dependency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r715231256
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,6 +61,7 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.12.2.jar
Review comment:
Dependencies should be updated here, https://github.com/apache/skywalking/blob/fdbbd5a94be8c196fa69056e1bacdd05e1041f31/dist-material/release-docs/LICENSE#L222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r742876708
##########
File path: tools/dependencies/known-oap-backend-dependencies.txt
##########
@@ -61,17 +61,26 @@ httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpcore-nio-4.4.13.jar
influxdb-java-2.15.jar
+iotdb-session-0.13.0-SNAPSHOT.jar
+iotdb-thrift-0.13.0-SNAPSHOT.jar
+istack-commons-runtime-4.0.0.jar
jackson-annotations-2.12.2.jar
jackson-core-2.12.2.jar
jackson-databind-2.12.2.jar
jackson-datatype-guava-2.9.10.jar
jackson-datatype-jdk8-2.9.10.jar
jackson-module-afterburner-2.12.2.jar
jackson-module-kotlin-2.8.8.jar
+jakarta.activation-2.0.0.jar
+jakarta.xml.bind-api-3.0.0.jar
Review comment:
I think it is your local cache that cause this issue, because when maven downloads a SNAPSHOT package it will firstly check whether it exists in your local repository (typically `~/.m2/repository`), and if exists it will be used directly, but SNAPSHOT can be overrode in remote repository so you might be using an old revision, as I check the latest revision of this iotdb-session SNAPSHOT is `iotdb-session-0.13.0-20211102.115555-1.jar Tue Nov 02 14:07:09 UTC 2021 67297` so you might want to update the local SNAPSHOT and repackage to see the changes, generally adding `-U` will do the trick (`./mvnw -U ....`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713753657
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBMetadataQueryDAO.java
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import com.google.gson.Gson;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import java.util.HashMap;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.util.StringUtil;
+import org.apache.skywalking.oap.server.core.analysis.NodeType;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.endpoint.EndpointTraffic;
+import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic;
+import org.apache.skywalking.oap.server.core.analysis.manual.service.ServiceTraffic;
+import org.apache.skywalking.oap.server.core.query.enumeration.Language;
+import org.apache.skywalking.oap.server.core.query.type.Attribute;
+import org.apache.skywalking.oap.server.core.query.type.Database;
+import org.apache.skywalking.oap.server.core.query.type.Endpoint;
+import org.apache.skywalking.oap.server.core.query.type.Service;
+import org.apache.skywalking.oap.server.core.query.type.ServiceInstance;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IMetadataQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+@Slf4j
+public class IoTDBMetadataQueryDAO implements IMetadataQueryDAO {
+ private static final Gson GSON = new Gson();
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<ServiceTraffic> serviceBuilder = new ServiceTraffic.Builder();
+ private final StorageHashMapBuilder<EndpointTraffic> endpointBuilder = new EndpointTraffic.Builder();
+ private final StorageHashMapBuilder<InstanceTraffic> instanceBuilder = new InstanceTraffic.Builder();
+
+ public IoTDBMetadataQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public List<Service> getAllServices(String group) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(ServiceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.NODE_TYPE_IDX, String.valueOf(NodeType.Normal.value()));
+ if (StringUtil.isNotEmpty(group)) {
+ indexAndValueMap.put(IoTDBClient.GROUP_IDX, group);
+ }
+ query = client.addQueryIndexValue(ServiceTraffic.INDEX_NAME, query, indexAndValueMap);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(ServiceTraffic.INDEX_NAME, query.toString(), serviceBuilder);
+ List<Service> serviceList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> serviceList.add(buildService((ServiceTraffic) storageData)));
+ return serviceList;
+ }
+
+ @Override
+ public List<Service> getAllBrowserServices() throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(ServiceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.NODE_TYPE_IDX, String.valueOf(NodeType.Browser.value()));
+ query = client.addQueryIndexValue(ServiceTraffic.INDEX_NAME, query, indexAndValueMap);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(ServiceTraffic.INDEX_NAME, query.toString(), serviceBuilder);
+ List<Service> serviceList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> serviceList.add(buildService((ServiceTraffic) storageData)));
+ return serviceList;
+ }
+
+ @Override
+ public List<Database> getAllDatabases() throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(ServiceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.NODE_TYPE_IDX, String.valueOf(NodeType.Database.value()));
+ query = client.addQueryIndexValue(ServiceTraffic.INDEX_NAME, query, indexAndValueMap);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(ServiceTraffic.INDEX_NAME, query.toString(), serviceBuilder);
+ List<Database> databaseList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> {
+ ServiceTraffic serviceTraffic = (ServiceTraffic) storageData;
+ Database database = new Database();
+ database.setId(serviceTraffic.id());
+ database.setName(serviceTraffic.getName());
+ databaseList.add(database);
+ });
+ return databaseList;
+ }
+
+ @Override
+ public List<Service> searchServices(String keyword) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(ServiceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.NODE_TYPE_IDX, String.valueOf(NodeType.Normal.value()));
+ query = client.addQueryIndexValue(ServiceTraffic.INDEX_NAME, query, indexAndValueMap);
+ if (!Strings.isNullOrEmpty(keyword)) {
+ query.append(" where ").append(ServiceTraffic.NAME).append(" like '%").append(keyword).append("%'");
+ }
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+ List<? super StorageData> storageDataList = client.filterQuery(ServiceTraffic.INDEX_NAME, query.toString(), serviceBuilder);
+ List<Service> serviceList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> serviceList.add(buildService((ServiceTraffic) storageData)));
+ return serviceList;
+ }
+
+ @Override
+ public Service searchService(String serviceCode) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(ServiceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.NODE_TYPE_IDX, String.valueOf(NodeType.Normal.value()));
+ query = client.addQueryIndexValue(ServiceTraffic.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where ").append(ServiceTraffic.NAME).append(" = \"").append(serviceCode).append("\"")
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+ List<? super StorageData> storageDataList = client.filterQuery(ServiceTraffic.INDEX_NAME, query.toString(), serviceBuilder);
+ return buildService((ServiceTraffic) storageDataList.get(0));
+ }
+
+ @Override
+ public List<Endpoint> searchEndpoint(String keyword, String serviceId, int limit) throws IOException {
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(EndpointTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ query = client.addQueryIndexValue(EndpointTraffic.INDEX_NAME, query, indexAndValueMap);
+ if (!Strings.isNullOrEmpty(keyword)) {
+ query.append(" where ").append(EndpointTraffic.NAME).append(" like '%").append(keyword).append("%'");
+ }
+ query.append(" limit ").append(limit).append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(EndpointTraffic.INDEX_NAME, query.toString(), endpointBuilder);
+ List<Endpoint> endpointList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> {
+ EndpointTraffic endpointTraffic = (EndpointTraffic) storageData;
+ Endpoint endpoint = new Endpoint();
+ endpoint.setId(endpointTraffic.id());
+ endpoint.setName(endpointTraffic.getName());
+ endpointList.add(endpoint);
+ });
+ return endpointList;
+ }
+
+ @Override
+ public List<ServiceInstance> getServiceInstances(long startTimestamp, long endTimestamp, String serviceId) throws IOException {
+ final long minuteTimeBucket = TimeBucket.getMinuteTimeBucket(startTimestamp);
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(InstanceTraffic.INDEX_NAME);
+ Map<String, String> indexAndValueMap = new HashMap<>();
+ indexAndValueMap.put(IoTDBClient.SERVICE_ID_IDX, serviceId);
+ query = client.addQueryIndexValue(InstanceTraffic.INDEX_NAME, query, indexAndValueMap);
+ query.append(" where ").append(InstanceTraffic.LAST_PING_TIME_BUCKET).append(" >= ").append(minuteTimeBucket)
+ .append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(InstanceTraffic.INDEX_NAME, query.toString(), instanceBuilder);
+ List<ServiceInstance> serviceInstanceList = new ArrayList<>(storageDataList.size());
+ storageDataList.forEach(storageData -> {
+ InstanceTraffic instanceTraffic = (InstanceTraffic) storageData;
+ ServiceInstance serviceInstance = new ServiceInstance();
+ serviceInstance.setId(instanceTraffic.id());
+ serviceInstance.setName(instanceTraffic.getName());
+ serviceInstance.setInstanceUUID(serviceInstance.getId());
+
+ JsonObject properties = instanceTraffic.getProperties();
+ if (!properties.isJsonNull()) {
+ for (Map.Entry<String, JsonElement> property : properties.entrySet()) {
+ String key = property.getKey();
+ String value = property.getValue().getAsString();
+ if (key.equals(InstanceTraffic.PropertyUtil.LANGUAGE)) {
+ serviceInstance.setLanguage(Language.value(value));
+ } else {
+ serviceInstance.getAttributes().add(new Attribute(key, value));
Review comment:
It has been initialized in constructor and won't be null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r714650302
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/base/IoTDBMetricsDAOTest.java
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.base;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.skywalking.oap.server.core.analysis.DownSampling;
+import org.apache.skywalking.oap.server.core.analysis.FunctionCategory;
+import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic;
+import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.model.ExtraQueryIndex;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBStorageConfig;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBTableMetaInfo;
+import org.junit.Before;
+import org.junit.Test;
+
+import static org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClientTest.retrieval;
+
+public class IoTDBMetricsDAOTest {
Review comment:
SkyWalking doesn't trust maven IT(not personal or relate to this PR, just general idea), SkyWalking is too complex, it is always better to have real OAP up and running, then verify logic.
IT in maven covers the API level check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713725404
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/pom.xml
##########
@@ -0,0 +1,55 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>server-storage-plugin</artifactId>
+ <groupId>org.apache.skywalking</groupId>
+ <version>8.8.0-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>storage-iotdb-plugin</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.iotdb</groupId>
+ <artifactId>iotdb-session</artifactId>
+ <version>0.12.2</version>
Review comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713752944
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBEventQueryDAO.java
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import org.apache.skywalking.oap.server.core.query.PaginationUtils;
+import org.apache.skywalking.oap.server.core.query.enumeration.Order;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.type.event.EventQueryCondition;
+import org.apache.skywalking.oap.server.core.query.type.event.EventType;
+import org.apache.skywalking.oap.server.core.query.type.event.Events;
+import org.apache.skywalking.oap.server.core.query.type.event.Source;
+import org.apache.skywalking.oap.server.core.source.Event;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+public class IoTDBEventQueryDAO implements IEventQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<Event> storageBuilder = new Event.Builder();
+
+ public IoTDBEventQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public Events queryEvents(EventQueryCondition condition) throws Exception {
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ StringBuilder query = new StringBuilder();
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(Event.INDEX_NAME);
+ query = client.addQueryAsterisk(Event.INDEX_NAME, query);
+ query = whereSQL(condition, query);
+ query.append(IoTDBClient.ALIGN_BY_DEVICE);
+
+ List<? super StorageData> storageDataList = client.filterQuery(Event.INDEX_NAME, query.toString(), storageBuilder);
+ final Events events = new Events();
+ int limitCount = 0;
+ PaginationUtils.Page page = PaginationUtils.INSTANCE.exchange(condition.getPaging());
+ for (int i = 0; i < storageDataList.size(); i++) {
+ if (i >= page.getFrom() && limitCount < page.getLimit()) {
+ limitCount++;
+ Event event = (Event) storageDataList.get(i);
+ org.apache.skywalking.oap.server.core.query.type.event.Event resultEvent = parseEvent(event);
+ events.getEvents().add(resultEvent);
+ }
+ }
+ events.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ final Order order = Objects.isNull(condition.getOrder()) ? Order.DES : condition.getOrder();
+ if (Order.DES.equals(order)) {
+ events.getEvents().sort(
+ (org.apache.skywalking.oap.server.core.query.type.event.Event e1,
+ org.apache.skywalking.oap.server.core.query.type.event.Event e2)
+ -> Long.compare(e2.getStartTime(), e1.getStartTime()));
+ } else {
+ events.getEvents().sort(
Review comment:
It has been initialized in constructor and won't be null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] LIU-WEI-git commented on a change in pull request #7766: IoTDB storage plugin
Posted by GitBox <gi...@apache.org>.
LIU-WEI-git commented on a change in pull request #7766:
URL: https://github.com/apache/skywalking/pull/7766#discussion_r713751264
##########
File path: oap-server/server-storage-plugin/storage-iotdb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/iotdb/query/IoTDBAlarmQueryDAO.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.iotdb.query;
+
+import com.google.common.base.Strings;
+import java.io.IOException;
+import java.util.List;
+import java.util.Objects;
+import org.apache.skywalking.oap.server.core.alarm.AlarmRecord;
+import org.apache.skywalking.oap.server.core.analysis.TimeBucket;
+import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag;
+import org.apache.skywalking.oap.server.core.query.enumeration.Scope;
+import org.apache.skywalking.oap.server.core.query.type.AlarmMessage;
+import org.apache.skywalking.oap.server.core.query.type.Alarms;
+import org.apache.skywalking.oap.server.core.storage.StorageData;
+import org.apache.skywalking.oap.server.core.storage.StorageHashMapBuilder;
+import org.apache.skywalking.oap.server.core.storage.query.IAlarmQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.storage.plugin.iotdb.IoTDBClient;
+
+public class IoTDBAlarmQueryDAO implements IAlarmQueryDAO {
+ private final IoTDBClient client;
+ private final StorageHashMapBuilder<AlarmRecord> storageBuilder = new AlarmRecord.Builder();
+
+ public IoTDBAlarmQueryDAO(IoTDBClient client) {
+ this.client = client;
+ }
+
+ @Override
+ public Alarms getAlarm(Integer scopeId, String keyword, int limit, int from, long startTB, long endTB, List<Tag> tags) throws IOException {
+ StringBuilder query = new StringBuilder();
+ // This method maybe have poor efficiency. It queries all data which meets a condition without select function.
+ // https://github.com/apache/iotdb/discussions/3888
+ query.append("select * from ").append(client.getStorageGroup()).append(IoTDBClient.DOT).append(AlarmRecord.INDEX_NAME);
+ query = client.addQueryAsterisk(AlarmRecord.INDEX_NAME, query);
+ query.append(" where 1=1");
+ if (Objects.nonNull(scopeId)) {
+ query.append(" and ").append(AlarmRecord.SCOPE).append(" = ").append(scopeId);
+ }
+ if (startTB != 0 && endTB != 0) {
+ query.append(" and ").append(IoTDBClient.TIME).append(" >= ").append(TimeBucket.getTimestamp(startTB));
+ query.append(" and ").append(IoTDBClient.TIME).append(" <= ").append(TimeBucket.getTimestamp(endTB));
+ }
+ if (!Strings.isNullOrEmpty(keyword)) {
+ query.append(" and ").append(AlarmRecord.ALARM_MESSAGE).append(" like '%").append(keyword).append("%'");
+ }
+ if (CollectionUtils.isNotEmpty(tags)) {
+ for (final Tag tag : tags) {
+ query.append(" and ").append(tag.getKey()).append(" = \"").append(tag.getValue()).append("\"");
+ }
+ }
+ // IoTDB doesn't support the query contains "1=1" and "*" at the meantime.
+ String queryString = query.toString().replace("1=1 and ", "");
+ queryString = queryString + IoTDBClient.ALIGN_BY_DEVICE;
+
+ Alarms alarms = new Alarms();
+ List<? super StorageData> storageDataList = client.filterQuery(AlarmRecord.INDEX_NAME, queryString, storageBuilder);
+ int limitCount = 0;
+ for (int i = 0; i < storageDataList.size(); i++) {
+ if (i >= from && limitCount < limit) {
+ limitCount++;
+ AlarmRecord alarmRecord = (AlarmRecord) storageDataList.get(i);
+ alarms.getMsgs().add(parseMessage(alarmRecord));
+ }
+ }
+ alarms.setTotal(storageDataList.size());
+ // resort by self, because of the select query result order by time.
+ alarms.getMsgs().sort((AlarmMessage m1, AlarmMessage m2) -> Long.compare(m2.getStartTime(), m1.getStartTime()));
Review comment:
It has been initialized in constructor and won't be null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org