You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by dmitriyHavrilovich <gi...@git.apache.org> on 2017/03/07 16:34:09 UTC
[GitHub] drill pull request #774: OpenTSDB storage plugin
GitHub user dmitriyHavrilovich opened a pull request:
https://github.com/apache/drill/pull/774
OpenTSDB storage plugin
Implementation of TSDB storage plugin. Plugin uses REST API to work with TSDB. Supported queries for now are listed below:
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mapr-demos/drill master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/774.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #774
----
commit 83cb33523f42b0abf9e5dfed6b1ee104fe3e1bcf
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2016-11-25T18:28:02Z
openTSDB plugin: initial commit
commit 20813645fa72513066c041fdb9ab93aa540fc6f5
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2016-12-02T12:31:42Z
Implement getting table names from openTSDB
commit 319c2d2f21d30cc70d1b183470ea8315d3a5381b
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2016-12-19T11:51:15Z
initial commit, add basic structure
commit 7b94619aca525a54e612708f93fac71dc71d5281
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2016-12-27T07:37:50Z
Partialy implement SELECT from openTSDB
commit c65e2e8f9aa973bbff76d37cc55ba2dc9bd5b5f2
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-01-12T20:51:04Z
Change table schema
commit f70bfbb14083fb1e1041f4481257a690eda2d61b
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-01-16T17:42:51Z
Implemented getting multiple rows from openTSDB table
commit d9643c220c21ce42e16aa21fea322898b130444a
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-01-17T14:06:58Z
Refactor Record Reader
commit 18cd35e53fa77de998e685b3834ec41666634a64
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-01-17T19:03:32Z
#1 Implemented aggregation function support
commit df81c967922fa4416d64d9e496a880f58b38c6bc
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-01-18T14:48:51Z
#4 Fixed issue with data duplication
commit 2505e7cbc885446971f3cb9f632ae4e16f6a8461
Author: Dmitriy Gavrilovich <dh...@cybervisiontech.com>
Date: 2017-01-31T12:44:26Z
Open TSDB plugin initial commit
commit a2019c9108c500e70b26b6839797f5f542ca6233
Author: dmitriyHavrilovich <dh...@cybervisiontech.com>
Date: 2017-01-31T23:31:29Z
Merge pull request #1 from mapr-demos/openTSDB-plugin
Open tsdb plugin
commit e85931c84839774d715effea6ec0dd56a0ba87a5
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-01T18:41:05Z
Change FROM syntax in query
commit 7ce213f44b880f4e4e4fe5567829a66c4e8ae82f
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-01T18:44:43Z
Fix formatting
commit 25f05816589c83f710c2fed18f3039530e97c797
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-03T18:50:02Z
Tag map divided into separate columns
commit 3deff2175fd34a456a3152c744e82959348fac09
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-06T14:57:55Z
#5 Add name validation
commit ffd2da2911b6131f8def9f290b6801fbe7b0589b
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-07T08:43:25Z
#6 Change kind of Exception to more clear form
commit 871ab712ee23809bd81132e2cca0ee2d6b8d6526
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-07T18:35:04Z
#7 Add parameters validation
commit f6387452497de2daabb2c95afe73d868ceb5188a
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-07T18:36:50Z
Refactor Schema
commit b44f861e2817be5aadc2e396d4444590e2e17ba1
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-08T16:17:29Z
#8 Add tests
commit 6fb8701daee7de5e2ac74805616340162fedc0e8
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-09T16:31:15Z
#8 Add printing and counting rows in tests
commit e5d5da107d6b376418a0ff4e352be024048031c0
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-15T14:12:01Z
Add downsample support
commit 2c0bf049842c79b05378c29c24cdebcfaeecbc37
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-15T15:35:45Z
Merge
commit 682179916948c028109d4de149c48324932ae21f
Author: Vlad Storona <vl...@users.noreply.github.com>
Date: 2017-02-15T15:42:39Z
Merge pull request #2 from mapr-demos/openTSDB-plugin
Open tsdb plugin
commit ad4cab46729a1f14269110c78803e4db4a7f3c5b
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-02-15T17:55:53Z
Change encoding to UTF_8 in test classes
commit bd53e6ac0db7ec7568657a83321fdbe41cc7b6b2
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-03-04T14:56:21Z
Fix reader, add POST API request
commit 2757d8903d0d2f4087c6b74f9b253e4f57acdc99
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-03-06T14:44:38Z
Fix timestamp
commit 5251482657a2dba43d244fb33b181419f1776484
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-03-06T16:03:36Z
Remove duplication in tables, fix downsample query
commit 16177ba76bd0df0c981c2597b40954cc489393b4
Author: Vlad Storona <vs...@cybervisiontech.com>
Date: 2017-03-06T19:35:25Z
Refactor RecordReader
commit 2f5831af343c68178528d31e8b779602b64cb14a
Author: Vlad Storona <vl...@users.noreply.github.com>
Date: 2017-03-06T19:58:41Z
Merge pull request #3 from mapr-demos/master
Merge pull request #2 from mapr-demos/openTSDB-plugin
commit e833d003182286ca51e2ab750c341492abd4878b
Author: Vlad Storona <vl...@users.noreply.github.com>
Date: 2017-03-06T20:03:55Z
Merge pull request #4 from mapr-demos/openTSDB-plugin
Open tsdb plugin
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148934012
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java ---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request to DB.
+ * <p>
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result set.
+ */
+public class DBQuery {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(DBQuery.class);
+ /**
+ * The start time for the query. This can be a relative or absolute timestamp.
+ */
+ private String start;
+ /**
+ * An end time for the query. If not supplied, the TSD will assume the local system time on the server.
+ * This may be a relative or absolute timestamp. This param is optional, and if it isn't specified we will send null
+ * to the db in this field, but in this case db will assume the local system time on the server.
+ */
+ private String end;
+ /**
+ * One or more sub subQueries used to select the time series to return.
+ */
+ private Set<Query> queries;
+
+ private DBQuery(Builder builder) {
+ this.start = builder.start;
+ this.end = builder.end;
+ this.queries = builder.queries;
+ }
+
+ public String getStart() {
+ return start;
+ }
+
+ public String getEnd() {
+ return end;
+ }
+
+ public Set<Query> getQueries() {
+ return queries;
+ }
+
+ public static class Builder {
+
+ private String start;
+ private String end;
+ private Set<Query> queries = new HashSet<>();
+
+ public Builder() {
+ }
+
+ public Builder setStartTime(String startTime) {
+ if (startTime == null) {
+ throw UserException.validationError()
+ .message("start param must be specified")
+ .build(log);
+ }
+ this.start = startTime;
+ return this;
+ }
+
+ public Builder setEndTime(String endTime) {
+ this.end = endTime;
+ return this;
+ }
+
+ public Builder setQueries(Set<Query> queries) {
+ if (queries.isEmpty()) {
+ throw UserException.validationError()
+ .message("Required params such as metric, aggregator weren't specified. " +
+ "Add this params to the query")
--- End diff --
`these`
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957508
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/Schema.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.*;
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATED_VALUE;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATE_TAGS;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.METRIC;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.TIMESTAMP;
+
+/**
+ * Abstraction for representing structure of openTSDB table
+ */
+public class Schema {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(Schema.class);
+
+ private final List<ColumnDTO> columns = new ArrayList<>();
+ private final Service db;
+ private final String name;
+
+ public Schema(Service db, String name) {
+ this.db = db;
+ this.name = name;
+ setupStructure();
+ }
+
+ private void setupStructure() {
+ columns.add(new ColumnDTO(METRIC.toString(), OpenTSDBTypes.STRING));
+ columns.add(new ColumnDTO(AGGREGATE_TAGS.toString(), OpenTSDBTypes.STRING));
+ columns.add(new ColumnDTO(TIMESTAMP.toString(), OpenTSDBTypes.TIMESTAMP));
+ columns.add(new ColumnDTO(AGGREGATED_VALUE.toString(), OpenTSDBTypes.DOUBLE));
+ columns.addAll(db.getUnfixedColumns(getParamsForQuery()));
+ }
+
+ /**
+ * Return list with all columns names and its types
+ *
+ * @return List<ColumnDTO>
+ */
+ public List<ColumnDTO> getColumns() {
+ return Collections.unmodifiableList(columns);
+ }
+
+ /**
+ * Number of columns in table
+ *
+ * @return number of table columns
+ */
+ public int getColumnCount() {
+ return columns.size();
+ }
+
+ /**
+ * @param columnIndex index of required column in table
+ * @return ColumnDTO
+ */
+ public ColumnDTO getColumnByIndex(int columnIndex) {
+ return columns.get(columnIndex);
+ }
+
+ // Create map with required params, for querying metrics.
+ // Without this params, we cannot make API request to db.
+ private HashMap<String, String> getParamsForQuery() {
--- End diff --
Ok.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567959
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+ private final DrillbitContext context;
+
+ private final OpenTSDBStoragePluginConfig engineConfig;
+ private final OpenTSDBSchemaFactory schemaFactory;
+
+ private final ServiceImpl db;
+
+ public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration, DrillbitContext context, String name) throws IOException {
+ this.context = context;
+ this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+ this.engineConfig = configuration;
+ this.db = new ServiceImpl("http://" + configuration.getConnection());
--- End diff --
Thank for this remark, I will move this choice to the config's.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569232
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
--- End diff --
Yes, missed it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957500
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Constants.java ---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+public interface Constants {
+ /**
+ * openTSDB required constants for API call
+ */
+ public static final String DEFAULT_TIME = "47y-ago";
--- End diff --
Yes, I remember that, but when we execute the query like `show tables;` drill must create the schema of the table. To create a schema we need to send a request to the db, but for this, we need the required params. But from this query, we can get only the metric name. That is why I use this default params.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146133055
--- Diff: distribution/pom.xml ---
@@ -190,11 +190,21 @@
</dependency>
<dependency>
<groupId>org.apache.drill.contrib</groupId>
+ <artifactId>drill-opentsdb-storage</artifactId>
--- End diff --
Why we have added `drill-opentsdb-storage` twice in this pom.xml?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568965
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
+ return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set<String> getAllTableNames() {
+ return getTableNames();
+ }
+
+ @Override
+ public List<ColumnDTO> getUnfixedColumnsToSchema() {
+ Set<MetricDTO> tables = getAllMetricsByTags();
+ List<ColumnDTO> unfixedColumns = new ArrayList<>();
+
+ for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+ if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+ }
+ }
+ }
+ return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+ if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+ } else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+ }
+ }
+
+ private Set<MetricDTO> getAllMetricsByTags() {
+ try {
+ return getAllMetricsFromDBByTags();
+ } catch (IOException e) {
+ logIOException(e);
--- End diff --
No, it is the critical exception. I will add exception throwing with the detailed message.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146134712
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
+ List<SchemaPath> projectedColumns) throws IOException {
+ setColumns(projectedColumns);
+ this.db = client;
+ this.subScanSpec = subScanSpec;
+ db.setupQueryParameters(subScanSpec.getTableName());
+ log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws ExecutionSetupException {
+ this.output = output;
+ Set<MetricDTO> tables = db.getTablesFromDB();
+ if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or it's empty", subScanSpec.getTableName()));
+ }
+ this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+ try {
+ return processOpenTSDBTablesData();
+ } catch (SchemaChangeException e) {
+ log.info(e.toString());
+ return 0;
+ }
+ }
+
+ @Override
+ protected boolean isSkipQuery() {
+ return super.isSkipQuery();
+ }
+
+ @Override
+ public void close() throws Exception {
+ }
+
+ static {
+ TYPES = ImmutableMap.<OpenTSDBTypes, MinorType>builder()
+ .put(OpenTSDBTypes.STRING, MinorType.VARCHAR)
+ .put(OpenTSDBTypes.DOUBLE, MinorType.FLOAT8)
+ .put(OpenTSDBTypes.TIMESTAMP, MinorType.TIMESTAMP)
+ .build();
+ }
+
+ private static class ProjectedColumnInfo {
+ ValueVector vv;
+ ColumnDTO openTSDBColumn;
+ }
+
+ private int processOpenTSDBTablesData() throws SchemaChangeException {
+ int rowCounter = 0;
+ while (tableIterator.hasNext()) {
+ MetricDTO metricDTO = tableIterator.next();
+ rowCounter = addRowResult(metricDTO, rowCounter);
+ }
+ return rowCounter;
+ }
+
+ private int addRowResult(MetricDTO table, int rowCounter) throws SchemaChangeException {
+ setupProjectedColsIfItNull();
+ for (String time : table.getDps().keySet()) {
+ String value = table.getDps().get(time);
+ setupDataToDrillTable(table, time, value, table.getTags(), rowCounter);
+ rowCounter++;
+ }
+ return rowCounter;
+ }
+
+ private void setupProjectedColsIfItNull() throws SchemaChangeException {
+ if (projectedCols == null) {
+ initCols(new Schema(db, subScanSpec.getTableName()));
+ }
+ }
+
+ private void setupDataToDrillTable(MetricDTO table, String timestamp, String value, Map<String, String> tags, int rowCount) {
+ for (ProjectedColumnInfo pci : projectedCols) {
+ switch (pci.openTSDBColumn.getColumnName()) {
+ case "metric":
+ setStringColumnValue(table.getMetric(), pci, rowCount);
+ break;
+ case "aggregate tags":
+ setStringColumnValue(table.getAggregateTags().toString(), pci, rowCount);
+ break;
+ case "timestamp":
+ setTimestampColumnValue(timestamp, pci, rowCount);
+ break;
+ case "aggregated value":
+ setDoubleColumnValue(value, pci, rowCount);
+ break;
+ default:
+ setStringColumnValue(tags.get(pci.openTSDBColumn.getColumnName()), pci, rowCount);
+ }
+ }
+ }
+
+ private void setTimestampColumnValue(String timestamp, ProjectedColumnInfo pci, int rowCount) {
+ setTimestampColumnValue(timestamp != null ? Long.parseLong(timestamp) : Long.parseLong("0"), pci, rowCount);
+ }
+
+ private void setDoubleColumnValue(String value, ProjectedColumnInfo pci, int rowCount) {
+ setDoubleColumnValue(value != null ? Double.parseDouble(value) : 0.0, pci, rowCount);
+ }
+
+ private void setStringColumnValue(String data, ProjectedColumnInfo pci, int rowCount) {
+ if (data == null) {
+ data = "null";
+ }
+ ByteBuffer value = ByteBuffer.wrap(data.getBytes(UTF_8));
+ ((NullableVarCharVector.Mutator) pci.vv.getMutator())
+ .setSafe(rowCount, value, 0, value.remaining());
+ }
+
+ private void setTimestampColumnValue(Long data, ProjectedColumnInfo pci, int rowCount) {
+ ((NullableTimeStampVector.Mutator) pci.vv.getMutator())
+ .setSafe(rowCount, data * 1000);
+ }
+
+ private void setDoubleColumnValue(Double data, ProjectedColumnInfo pci, int rowCount) {
+ ((NullableFloat8Vector.Mutator) pci.vv.getMutator())
+ .setSafe(rowCount, data);
+ }
+
+ private void initCols(Schema schema) throws SchemaChangeException {
+ ImmutableList.Builder<ProjectedColumnInfo> pciBuilder = ImmutableList.builder();
+
+ for (int i = 0; i < schema.getColumnCount(); i++) {
+
+ ColumnDTO column = schema.getColumnByIndex(i);
+ final String name = column.getColumnName();
+ final OpenTSDBTypes type = column.getColumnType();
+ TypeProtos.MinorType minorType = TYPES.get(type);
+
+ if (isMinorTypeNull(minorType)) {
+ logExceptionMessage(name, type);
--- End diff --
Why don't we fail instead when we don't support data type so user know we can not query the data rather then just skipping?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569476
--- Diff: contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestBase.java ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+import org.apache.drill.PlanTestBase;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.rpc.user.QueryDataBatch;
+import org.junit.Assert;
+
+import java.util.List;
+
+class TestBase extends PlanTestBase {
--- End diff --
Ok.
---
[GitHub] drill issue #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on the issue:
https://github.com/apache/drill/pull/774
@arina-ielchiieva Thanks for the review. I updated the PR.
Please review updated changes.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147584453
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+ private final Map<String, OpenTSDBDatabaseSchema> schemaMap = Maps.newHashMap();
+
+ OpenTSDBTables(String name) {
+ super(Collections.<String>emptyList(), name);
+ }
+
+ @Override
+ public AbstractSchema getSubSchema(String name) {
+ Set<String> tables;
+ if (!schemaMap.containsKey(name)) {
+ tables = plugin.getClient().getAllTableNames();
+ schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, name));
+ }
+ return schemaMap.get(name);
+ }
+
+ @Override
+ public Set<String> getSubSchemaNames() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+ return new DrillOpenTSDBTable(schemaName, plugin, new Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+ log.warn("Failure while retrieving openTSDB table {}", name, e);
+ return null;
+ }
+ }
+
+ @Override
+ public Set<String> getTableNames() {
+ return plugin.getClient().getAllTableNames();
+ }
+
+ @Override
+ public CreateTableEntry createNewTable(final String tableName, List<String> partitionColumns) {
+ return null;
+ }
+
+ @Override
+ public void dropTable(String tableName) {
--- End diff --
Parent behavior is to fail indication that table dropping is not allowed. Why to override with empty method?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569400
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+ private final Map<String, OpenTSDBDatabaseSchema> schemaMap = Maps.newHashMap();
+
+ OpenTSDBTables(String name) {
+ super(Collections.<String>emptyList(), name);
+ }
+
+ @Override
+ public AbstractSchema getSubSchema(String name) {
+ Set<String> tables;
+ if (!schemaMap.containsKey(name)) {
+ tables = plugin.getClient().getAllTableNames();
+ schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, name));
+ }
+ return schemaMap.get(name);
+ }
+
+ @Override
+ public Set<String> getSubSchemaNames() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+ return new DrillOpenTSDBTable(schemaName, plugin, new Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+ log.warn("Failure while retrieving openTSDB table {}", name, e);
+ return null;
+ }
+ }
+
+ @Override
+ public Set<String> getTableNames() {
+ return plugin.getClient().getAllTableNames();
+ }
+
+ @Override
+ public CreateTableEntry createNewTable(final String tableName, List<String> partitionColumns) {
+ return null;
+ }
+
+ @Override
+ public void dropTable(String tableName) {
--- End diff --
Ok, I will remove this overriding of the method.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147583993
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
+ return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set<String> getAllTableNames() {
+ return getTableNames();
+ }
+
+ @Override
+ public List<ColumnDTO> getUnfixedColumnsToSchema() {
+ Set<MetricDTO> tables = getAllMetricsByTags();
+ List<ColumnDTO> unfixedColumns = new ArrayList<>();
+
+ for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+ if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+ }
+ }
+ }
+ return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+ if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+ } else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+ }
+ }
+
+ private Set<MetricDTO> getAllMetricsByTags() {
+ try {
+ return getAllMetricsFromDBByTags();
+ } catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<String> getTableNames() {
+ try {
+ return client.getAllTablesName().execute().body();
+ } catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<MetricDTO> getAllTablesWithSpecialTag(DBQuery base) throws IOException {
+ return client.getTables(base).execute().body();
+ }
+
+ private Set<MetricDTO> getAllMetricsFromDBByTags() throws IOException {
+ Map<String, String> tags = new HashMap<>();
+
+ Query subQuery = new Query.Builder(queryParameters.get(METRIC))
+ .setAggregator(queryParameters.get(AGGREGATOR))
+ .setDownsample(queryParameters.get(DOWNSAMPLE))
+ .setTags(tags).build();
+
+ Set<Query> queries = new HashSet<>();
+ queries.add(subQuery);
+
+ DBQuery baseQuery = new DBQuery.Builder()
+ .setQueries(queries)
+ .build();
+
+ Set<MetricDTO> tables =
+ getBaseTables(baseQuery);
+
+ Set<String> extractedTags =
+ getTagsFromTables(tables);
+
+ tables.clear();
+
+ for (String value : extractedTags) {
+ transformTagsForRequest(tags, value);
--- End diff --
For each extracted tag you will call method `transformTagsForRequest` where you take `tags` map and clear it each time, seems a bit of confusing...
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567810
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
+ List<SchemaPath> projectedColumns) throws IOException {
+ setColumns(projectedColumns);
+ this.db = client;
+ this.subScanSpec = subScanSpec;
+ db.setupQueryParameters(subScanSpec.getTableName());
+ log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws ExecutionSetupException {
+ this.output = output;
+ Set<MetricDTO> tables = db.getTablesFromDB();
+ if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or it's empty", subScanSpec.getTableName()));
--- End diff --
Ok, this is really weird behaviour, I will change it. Yes, `db.getTablesFromDB();` returns all the metrics from db. I will add comments to the service methods.
Also, I completely agree that the exception handling in this place really confusing, I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567693
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List<SchemaPath> columns;
+ private List<OpenTSDBWork> openTSDBWorkList = Lists.newArrayList();
+ private List<EndpointAffinity> affinities;
+
+ private boolean filterPushedDown = false;
+
+ @JsonCreator
+ public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec") OpenTSDBScanSpec openTSDBScanSpec,
+ @JsonProperty("storage") OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+ @JsonProperty("columns") List<SchemaPath> columns,
+ @JacksonInject StoragePluginRegistry pluginRegistry) throws IOException, ExecutionSetupException {
+ this((OpenTSDBStoragePlugin) pluginRegistry.getPlugin(openTSDBStoragePluginConfig), openTSDBScanSpec, columns);
+ }
+
+ public OpenTSDBGroupScan(OpenTSDBStoragePlugin storagePlugin,
+ OpenTSDBScanSpec scanSpec, List<SchemaPath> columns) {
+ super((String) null);
+ this.storagePlugin = storagePlugin;
+ this.storagePluginConfig = storagePlugin.getConfig();
+ this.openTSDBScanSpec = scanSpec;
+ this.columns = columns == null || columns.size() == 0 ? ALL_COLUMNS : columns;
+ init();
+ }
+
+ /**
+ * Private constructor, used for cloning.
+ *
+ * @param that The OpenTSDBGroupScan to clone
+ */
+ private OpenTSDBGroupScan(OpenTSDBGroupScan that) {
+ super((String) null);
+ this.columns = that.columns;
+ this.openTSDBScanSpec = that.openTSDBScanSpec;
+ this.storagePlugin = that.storagePlugin;
+ this.storagePluginConfig = that.storagePluginConfig;
+ this.filterPushedDown = that.filterPushedDown;
+ this.openTSDBWorkList = that.openTSDBWorkList;
+ this.assignments = that.assignments;
+ init();
+ }
+
+ private void init() {
+ Collection<DrillbitEndpoint> endpoints = storagePlugin.getContext().getBits();
+ Map<String, DrillbitEndpoint> endpointMap = Maps.newHashMap();
+
+ for (DrillbitEndpoint endpoint : endpoints) {
+ endpointMap.put(endpoint.getAddress(), endpoint);
+ }
+ }
+
+ @Override
+ public List<EndpointAffinity> getOperatorAffinity() {
+ if (affinities == null) {
+ affinities = AffinityCreator.getAffinityMap(openTSDBWorkList);
+ }
+ return affinities;
+ }
+
+ @Override
+ public int getMaxParallelizationWidth() {
+ return openTSDBWorkList.size();
+ }
+
+ @Override
+ public void applyAssignments(List<DrillbitEndpoint> incomingEndpoints) {
+ assignments = AssignmentCreator.getMappings(incomingEndpoints, openTSDBWorkList);
+ }
+
+ @Override
+ public OpenTSDBSubScan getSpecificScan(int minorFragmentId) {
+ List<OpenTSDBSubScanSpec> scanSpecList = Lists.newArrayList();
+ scanSpecList.add(new OpenTSDBSubScanSpec(getTableName()));
+ return new OpenTSDBSubScan(storagePlugin, storagePluginConfig, scanSpecList, this.columns);
+ }
+
+ @Override
+ public ScanStats getScanStats() {
+ //magic number ?
+ long recordCount = 100000L * openTSDBWorkList.size();
--- End diff --
It is a weird calculation, I will rewrite it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567569
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Constants.java ---
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+public class Constants {
--- End diff --
Ok, I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567715
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List<SchemaPath> columns;
+ private List<OpenTSDBWork> openTSDBWorkList = Lists.newArrayList();
+ private List<EndpointAffinity> affinities;
+
+ private boolean filterPushedDown = false;
+
+ @JsonCreator
+ public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec") OpenTSDBScanSpec openTSDBScanSpec,
+ @JsonProperty("storage") OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+ @JsonProperty("columns") List<SchemaPath> columns,
+ @JacksonInject StoragePluginRegistry pluginRegistry) throws IOException, ExecutionSetupException {
+ this((OpenTSDBStoragePlugin) pluginRegistry.getPlugin(openTSDBStoragePluginConfig), openTSDBScanSpec, columns);
+ }
+
+ public OpenTSDBGroupScan(OpenTSDBStoragePlugin storagePlugin,
+ OpenTSDBScanSpec scanSpec, List<SchemaPath> columns) {
+ super((String) null);
+ this.storagePlugin = storagePlugin;
+ this.storagePluginConfig = storagePlugin.getConfig();
+ this.openTSDBScanSpec = scanSpec;
+ this.columns = columns == null || columns.size() == 0 ? ALL_COLUMNS : columns;
+ init();
+ }
+
+ /**
+ * Private constructor, used for cloning.
+ *
+ * @param that The OpenTSDBGroupScan to clone
+ */
+ private OpenTSDBGroupScan(OpenTSDBGroupScan that) {
+ super((String) null);
+ this.columns = that.columns;
+ this.openTSDBScanSpec = that.openTSDBScanSpec;
+ this.storagePlugin = that.storagePlugin;
+ this.storagePluginConfig = that.storagePluginConfig;
+ this.filterPushedDown = that.filterPushedDown;
+ this.openTSDBWorkList = that.openTSDBWorkList;
+ this.assignments = that.assignments;
+ init();
+ }
+
+ private void init() {
+ Collection<DrillbitEndpoint> endpoints = storagePlugin.getContext().getBits();
+ Map<String, DrillbitEndpoint> endpointMap = Maps.newHashMap();
+
+ for (DrillbitEndpoint endpoint : endpoints) {
+ endpointMap.put(endpoint.getAddress(), endpoint);
+ }
+ }
+
+ @Override
+ public List<EndpointAffinity> getOperatorAffinity() {
+ if (affinities == null) {
+ affinities = AffinityCreator.getAffinityMap(openTSDBWorkList);
+ }
+ return affinities;
+ }
+
+ @Override
+ public int getMaxParallelizationWidth() {
+ return openTSDBWorkList.size();
+ }
+
+ @Override
+ public void applyAssignments(List<DrillbitEndpoint> incomingEndpoints) {
+ assignments = AssignmentCreator.getMappings(incomingEndpoints, openTSDBWorkList);
+ }
+
+ @Override
+ public OpenTSDBSubScan getSpecificScan(int minorFragmentId) {
+ List<OpenTSDBSubScanSpec> scanSpecList = Lists.newArrayList();
+ scanSpecList.add(new OpenTSDBSubScanSpec(getTableName()));
+ return new OpenTSDBSubScan(storagePlugin, storagePluginConfig, scanSpecList, this.columns);
+ }
+
+ @Override
+ public ScanStats getScanStats() {
+ //magic number ?
+ long recordCount = 100000L * openTSDBWorkList.size();
+ return new ScanStats(ScanStats.GroupScanProperty.NO_EXACT_ROW_COUNT, recordCount, 1, recordCount);
+ }
+
+ @Override
+ @JsonIgnore
+ public PhysicalOperator getNewWithChildren(List<PhysicalOperator> children) {
+ Preconditions.checkArgument(children.isEmpty());
+ return new OpenTSDBGroupScan(this);
+ }
+
+ @Override
+ public String getDigest() {
+ return toString();
+ }
+
+ @Override
+ @JsonIgnore
+ public boolean canPushdownProjects(List<SchemaPath> columns) {
+ return true;
+ }
+
+ @JsonIgnore
+ public String getTableName() {
+ return getOpenTSDBScanSpec().getTableName();
+ }
+
+ @JsonProperty
+ public OpenTSDBScanSpec getOpenTSDBScanSpec() {
+ return openTSDBScanSpec;
+ }
+
+ @Override
+ public GroupScan clone(List<SchemaPath> columns) {
+ OpenTSDBGroupScan newScan = new OpenTSDBGroupScan(this);
+ newScan.columns = columns;
+ return newScan;
+ }
+
+ @Override
+ public String toString() {
+ return "OpenTSDBGroupScan [OpenTSDBScanSpec="
+ + openTSDBScanSpec + "]";
+ }
+
+ private static class OpenTSDBWork implements CompleteWork {
+
+ private EndpointByteMapImpl byteMap = new EndpointByteMapImpl();
+
+ @Override
+ public long getTotalBytes() {
+ return DEFAULT_TABLET_SIZE;
--- End diff --
Actually, I cannot explain how this variable was calculated. But I reworked `OpenTSDBGroupScan` and now this variable not required.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568059
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePluginConfig.java ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.drill.common.logical.StoragePluginConfigBase;
+
+import java.io.IOException;
+import java.util.Objects;
+
+@JsonTypeName(OpenTSDBStoragePluginConfig.NAME)
+public class OpenTSDBStoragePluginConfig extends StoragePluginConfigBase {
+
+ public static final String NAME = "openTSDB";
+
+ private final String connection;
+
+ @JsonCreator
+ public OpenTSDBStoragePluginConfig(@JsonProperty("connection") String connection) throws IOException {
+ this.connection = connection;
--- End diff --
Yes, we should. I will add null check.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146134351
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
--- End diff --
final?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147584018
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
+ return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set<String> getAllTableNames() {
+ return getTableNames();
+ }
+
+ @Override
+ public List<ColumnDTO> getUnfixedColumnsToSchema() {
+ Set<MetricDTO> tables = getAllMetricsByTags();
+ List<ColumnDTO> unfixedColumns = new ArrayList<>();
+
+ for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+ if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+ }
+ }
+ }
+ return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+ if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+ } else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+ }
+ }
+
+ private Set<MetricDTO> getAllMetricsByTags() {
+ try {
+ return getAllMetricsFromDBByTags();
+ } catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<String> getTableNames() {
+ try {
+ return client.getAllTablesName().execute().body();
+ } catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<MetricDTO> getAllTablesWithSpecialTag(DBQuery base) throws IOException {
+ return client.getTables(base).execute().body();
+ }
+
+ private Set<MetricDTO> getAllMetricsFromDBByTags() throws IOException {
+ Map<String, String> tags = new HashMap<>();
+
+ Query subQuery = new Query.Builder(queryParameters.get(METRIC))
+ .setAggregator(queryParameters.get(AGGREGATOR))
+ .setDownsample(queryParameters.get(DOWNSAMPLE))
+ .setTags(tags).build();
+
+ Set<Query> queries = new HashSet<>();
+ queries.add(subQuery);
+
+ DBQuery baseQuery = new DBQuery.Builder()
+ .setQueries(queries)
+ .build();
+
+ Set<MetricDTO> tables =
+ getBaseTables(baseQuery);
+
+ Set<String> extractedTags =
+ getTagsFromTables(tables);
+
+ tables.clear();
--- End diff --
Better use new object rather than clearing existing one.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146134172
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
+ List<SchemaPath> projectedColumns) throws IOException {
+ setColumns(projectedColumns);
+ this.db = client;
+ this.subScanSpec = subScanSpec;
+ db.setupQueryParameters(subScanSpec.getTableName());
+ log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws ExecutionSetupException {
+ this.output = output;
+ Set<MetricDTO> tables = db.getTablesFromDB();
+ if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or it's empty", subScanSpec.getTableName()));
--- End diff --
1. If we didn't find table, it's fine if we fail but why do we return error if table is empty? Should not we just return empty result? What `db.getTablesFromDB();` actually returns? All tables from the db?
2. I have incorrectly indicated connection to the opentsdb. And when I have run the query `SELECT * FROM openTSDB.(metric=mymetric.stock)` and got validation error, though the real root cause was that we could not connect to the opentsdb. In `drillbit.out` the following error was logged:
```
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:580)
at okhttp3.internal.Platform.connectSocket(Platform.java:121)
at okhttp3.internal.io.RealConnection.connectSocket(RealConnection.java:185)
at okhttp3.internal.io.RealConnection.buildConnection(RealConnection.java:170)
at okhttp3.internal.io.RealConnection.connect(RealConnection.java:111)
at okhttp3.internal.http.StreamAllocation.findConnection(StreamAllocation.java:187)
at okhttp3.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:123)
at okhttp3.internal.http.StreamAllocation.newStream(StreamAllocation.java:93)
at okhttp3.internal.http.HttpEngine.connect(HttpEngine.java:296)
at okhttp3.internal.http.HttpEngine.sendRequest(HttpEngine.java:248)
at okhttp3.RealCall.getResponse(RealCall.java:243)
at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:201)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:163)
at okhttp3.RealCall.execute(RealCall.java:57)
at retrofit2.OkHttpCall.execute(OkHttpCall.java:174)
at org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getTableNames(ServiceImpl.java:111)
at org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl.getAllTableNames(ServiceImpl.java:70)
at org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory$OpenTSDBTables.getSubSchema(OpenTSDBSchemaFactory.java:74)
at org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory$OpenTSDBTables.getSubSchema(OpenTSDBSchemaFactory.java:63)
at org.apache.calcite.jdbc.SimpleCalciteSchema.getSubSchema(SimpleCalciteSchema.java:112)
at org.apache.calcite.prepare.CalciteCatalogReader.getSchema(CalciteCatalogReader.java:160)
at org.apache.calcite.prepare.CalciteCatalogReader.getTableFrom(CalciteCatalogReader.java:113)
at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:102)
at org.apache.drill.exec.planner.sql.SqlConverter$DrillCalciteCatalogReader.getTable(SqlConverter.java:484)
at org.apache.drill.exec.planner.sql.SqlConverter$DrillCalciteCatalogReader.getTable(SqlConverter.java:429)
at org.apache.calcite.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:75)
at org.apache.calcite.sql.validate.DelegatingScope.getTableNamespace(DelegatingScope.java:124)
at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:104)
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:86)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:883)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:869)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2806)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2791)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3014)
at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:86)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:883)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:869)
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:210)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:843)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:557)
at org.apache.drill.exec.planner.sql.SqlConverter.validate(SqlConverter.java:170)
at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:617)
at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateAndConvert(DefaultSqlHandler.java:192)
at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:164)
at org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:131)
at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:79)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1048)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:748)
```
When I have executed `use opentsdb; show tables;` with incorrect configuration, I got empty result, though I was expecting error indicating that we could not connect to the db as well.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146134773
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
+ List<SchemaPath> projectedColumns) throws IOException {
+ setColumns(projectedColumns);
+ this.db = client;
+ this.subScanSpec = subScanSpec;
+ db.setupQueryParameters(subScanSpec.getTableName());
+ log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws ExecutionSetupException {
+ this.output = output;
+ Set<MetricDTO> tables = db.getTablesFromDB();
+ if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or it's empty", subScanSpec.getTableName()));
+ }
+ this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+ try {
+ return processOpenTSDBTablesData();
+ } catch (SchemaChangeException e) {
+ log.info(e.toString());
--- End diff --
I guess we should fail on `SchemaChangeException` indicating we don't support it rather then returning 0?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146135253
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+ private final DrillbitContext context;
+
+ private final OpenTSDBStoragePluginConfig engineConfig;
+ private final OpenTSDBSchemaFactory schemaFactory;
+
+ private final ServiceImpl db;
+
+ public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration, DrillbitContext context, String name) throws IOException {
+ this.context = context;
+ this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+ this.engineConfig = configuration;
+ this.db = new ServiceImpl("http://" + configuration.getConnection());
--- End diff --
Can it be `https`? May be user should be responsible of setting this in config?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147584562
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+ private final Map<String, OpenTSDBDatabaseSchema> schemaMap = Maps.newHashMap();
+
+ OpenTSDBTables(String name) {
+ super(Collections.<String>emptyList(), name);
+ }
+
+ @Override
+ public AbstractSchema getSubSchema(String name) {
+ Set<String> tables;
+ if (!schemaMap.containsKey(name)) {
+ tables = plugin.getClient().getAllTableNames();
+ schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, name));
+ }
+ return schemaMap.get(name);
+ }
+
+ @Override
+ public Set<String> getSubSchemaNames() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+ return new DrillOpenTSDBTable(schemaName, plugin, new Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+ log.warn("Failure while retrieving openTSDB table {}", name, e);
+ return null;
+ }
+ }
+
+ @Override
+ public Set<String> getTableNames() {
+ return plugin.getClient().getAllTableNames();
+ }
+
+ @Override
+ public CreateTableEntry createNewTable(final String tableName, List<String> partitionColumns) {
+ return null;
--- End diff --
Parent behavior is to fail indication that table creation is not allowed. I guess we can stick to it rather then returning null.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567585
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/DrillOpenTSDBTable.java ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+
+import java.util.List;
+
+public class DrillOpenTSDBTable extends DynamicDrillTable {
+
+ private final Schema schema;
+
+ public DrillOpenTSDBTable(String storageEngineName, OpenTSDBStoragePlugin plugin, Schema schema, OpenTSDBScanSpec scanSpec) {
+ super(plugin, storageEngineName, scanSpec);
+ this.schema = schema;
+ }
+
+ @Override
+ public RelDataType getRowType(final RelDataTypeFactory typeFactory) {
+ List<String> names = Lists.newArrayList();
+ List<RelDataType> types = Lists.newArrayList();
+ convertToRelDataType(typeFactory, names, types);
+ return typeFactory.createStructType(types, names);
+ }
+
+ private void convertToRelDataType(RelDataTypeFactory typeFactory, List<String> names, List<RelDataType> types) {
+ for (ColumnDTO column : schema.getColumns()) {
+ names.add(column.getColumnName());
+ RelDataType type = getSqlTypeFromOpenTSDBType(typeFactory, column.getColumnType());
+ type = typeFactory.createTypeWithNullability(type, column.isNullable());
+ types.add(type);
+ }
+ }
+
+ private RelDataType getSqlTypeFromOpenTSDBType(RelDataTypeFactory typeFactory, OpenTSDBTypes type) {
+ switch (type) {
+ case STRING:
+ return typeFactory.createSqlType(SqlTypeName.VARCHAR, Integer.MAX_VALUE);
+ case DOUBLE:
+ return typeFactory.createSqlType(SqlTypeName.DOUBLE);
+ case TIMESTAMP:
+ return typeFactory.createSqlType(SqlTypeName.TIMESTAMP);
+ default:
+ throw new UnsupportedOperationException("Unsupported type.");
--- End diff --
Sure, I will add information about unsupported types to the exception message.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569587
--- Diff: contrib/storage-opentsdb/src/test/resources/logback.xml ---
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8" ?>
--- End diff --
Ok, I will remove this file.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146134878
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+ private final Map<String, OpenTSDBDatabaseSchema> schemaMap = Maps.newHashMap();
+
+ OpenTSDBTables(String name) {
+ super(Collections.<String>emptyList(), name);
+ }
+
+ @Override
+ public AbstractSchema getSubSchema(String name) {
+ Set<String> tables;
+ if (!schemaMap.containsKey(name)) {
+ tables = plugin.getClient().getAllTableNames();
+ schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, name));
+ }
+ return schemaMap.get(name);
+ }
+
+ @Override
+ public Set<String> getSubSchemaNames() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+ return new DrillOpenTSDBTable(schemaName, plugin, new Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+ log.warn("Failure while retrieving openTSDB table {}", name, e);
--- End diff --
What will happen if we return null? Will user get meaningful error? Maybe we should fail here? Or at least log as error rather then warning?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957506
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Util.java ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.base.Splitter;
+
+import java.util.Map;
+
+public class Util {
+
+ /**
+ * Parse FROM parameters to Map representation
+ *
+ * @param rowData with this syntax (metric=warp.speed.test)
+ * @return Map with params key: metric, value: warp.speed.test
+ */
+ public static Map<String, String> parseFromRowData(String rowData) {
+ String FROMRowData = rowData.replaceAll("[()]", "");
--- End diff --
I will rename it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569362
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+ private final Map<String, OpenTSDBDatabaseSchema> schemaMap = Maps.newHashMap();
+
+ OpenTSDBTables(String name) {
+ super(Collections.<String>emptyList(), name);
+ }
+
+ @Override
+ public AbstractSchema getSubSchema(String name) {
+ Set<String> tables;
+ if (!schemaMap.containsKey(name)) {
+ tables = plugin.getClient().getAllTableNames();
+ schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, name));
+ }
+ return schemaMap.get(name);
+ }
+
+ @Override
+ public Set<String> getSubSchemaNames() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+ return new DrillOpenTSDBTable(schemaName, plugin, new Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+ log.warn("Failure while retrieving openTSDB table {}", name, e);
--- End diff --
I think we must throw a meaningful exception to the user. I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957511
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java ---
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request to DB.
+ * <p>
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result set.
+ */
+public class DBQuery {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(DBQuery.class);
+ /**
+ * The start time for the query. This can be a relative or absolute timestamp.
+ */
+ private String start;
+ /**
+ * An end time for the query. If not supplied, the TSD will assume the local system time on the server.
+ * This may be a relative or absolute timestamp. This param is optional, and if it isn't specified we will send null
+ * to the db in this field, but in this case db will assume the local system time on the server.
+ */
+ private String end;
+ /**
+ * One or more sub subQueries used to select the time series to return.
+ */
+ private Set<Query> queries;
+
+ private DBQuery(Builder builder) {
+ this.start = builder.start;
+ this.end = builder.end;
+ this.queries = builder.queries;
+ }
+
+ public String getStart() {
+ return start;
+ }
+
+ public String getEnd() {
+ return end;
+ }
+
+ public Set<Query> getQueries() {
+ return queries;
+ }
+
+ public static class Builder {
+
+ private String start;
+ private String end;
+ private Set<Query> queries = new HashSet<>();
+
+ public Builder() {
+ }
+
+ public Builder setStartTime(String startTime) {
+ if (startTime == null) {
+ throw UserException.validationError()
+ .message("start param must be specified")
+ .build(log);
+ }
+ this.start = startTime;
+ return this;
+ }
+
+ public Builder setEndTime(String endTime) {
+ this.end = endTime;
+ return this;
+ }
+
+ public Builder setQueries(Set<Query> queries) {
+ if (queries.isEmpty()) {
+ throw UserException.validationError()
+ .message("Required params such as metric, aggregator weren't specified. " +
+ "Add this params to the query")
--- End diff --
Ok
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146135440
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List<SchemaPath> columns;
+ private List<OpenTSDBWork> openTSDBWorkList = Lists.newArrayList();
+ private List<EndpointAffinity> affinities;
+
+ private boolean filterPushedDown = false;
+
+ @JsonCreator
+ public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec") OpenTSDBScanSpec openTSDBScanSpec,
+ @JsonProperty("storage") OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+ @JsonProperty("columns") List<SchemaPath> columns,
--- End diff --
How this class is deserialized if you don't have getters for `storage` and `columns`?
One of the options to test serde is to create physical plan for your query and then submit physical plan (similar approach was used in https://github.com/apache/drill/pull/1014).
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146135479
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/DrillOpenTSDBTable.java ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+
+import java.util.List;
+
+public class DrillOpenTSDBTable extends DynamicDrillTable {
+
+ private final Schema schema;
+
+ public DrillOpenTSDBTable(String storageEngineName, OpenTSDBStoragePlugin plugin, Schema schema, OpenTSDBScanSpec scanSpec) {
+ super(plugin, storageEngineName, scanSpec);
+ this.schema = schema;
+ }
+
+ @Override
+ public RelDataType getRowType(final RelDataTypeFactory typeFactory) {
+ List<String> names = Lists.newArrayList();
+ List<RelDataType> types = Lists.newArrayList();
+ convertToRelDataType(typeFactory, names, types);
+ return typeFactory.createStructType(types, names);
+ }
+
+ private void convertToRelDataType(RelDataTypeFactory typeFactory, List<String> names, List<RelDataType> types) {
+ for (ColumnDTO column : schema.getColumns()) {
+ names.add(column.getColumnName());
+ RelDataType type = getSqlTypeFromOpenTSDBType(typeFactory, column.getColumnType());
+ type = typeFactory.createTypeWithNullability(type, column.isNullable());
+ types.add(type);
+ }
+ }
+
+ private RelDataType getSqlTypeFromOpenTSDBType(RelDataTypeFactory typeFactory, OpenTSDBTypes type) {
+ switch (type) {
+ case STRING:
+ return typeFactory.createSqlType(SqlTypeName.VARCHAR, Integer.MAX_VALUE);
+ case DOUBLE:
+ return typeFactory.createSqlType(SqlTypeName.DOUBLE);
+ case TIMESTAMP:
+ return typeFactory.createSqlType(SqlTypeName.TIMESTAMP);
+ default:
+ throw new UnsupportedOperationException("Unsupported type.");
--- End diff --
I guess it's better to add error message indicating which type is unsupported and which types are currently supported.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568739
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
+public class OpenTSDBSubScan extends AbstractBase implements SubScan {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(OpenTSDBSubScan.class);
+
+ @JsonProperty
+ public final OpenTSDBStoragePluginConfig storage;
+
+ private final List<SchemaPath> columns;
+ private final OpenTSDBStoragePlugin openTSDBStoragePlugin;
+ private final List<OpenTSDBSubScanSpec> tabletScanSpecList;
+
+ @JsonCreator
+ public OpenTSDBSubScan(@JacksonInject StoragePluginRegistry registry,
+ @JsonProperty("storage") StoragePluginConfig storage,
+ @JsonProperty("tabletScanSpecList") LinkedList<OpenTSDBSubScanSpec> tabletScanSpecList,
+ @JsonProperty("columns") List<SchemaPath> columns) throws ExecutionSetupException {
+ super((String) null);
+ openTSDBStoragePlugin = (OpenTSDBStoragePlugin) registry.getPlugin(storage);
+ this.tabletScanSpecList = tabletScanSpecList;
+ this.storage = (OpenTSDBStoragePluginConfig) storage;
+ this.columns = columns;
+ }
+
+ public OpenTSDBSubScan(OpenTSDBStoragePlugin plugin, OpenTSDBStoragePluginConfig config,
+ List<OpenTSDBSubScanSpec> tabletInfoList, List<SchemaPath> columns) {
+ super((String) null);
+ openTSDBStoragePlugin = plugin;
+ storage = config;
+ this.tabletScanSpecList = tabletInfoList;
+ this.columns = columns;
+ }
+
+ @Override
+ public int getOperatorType() {
+ return 0;
+ }
+
+ @Override
+ public boolean isExecutable() {
+ return false;
+ }
+
+ @Override
+ public PhysicalOperator getNewWithChildren(List<PhysicalOperator> children) throws ExecutionSetupException {
+ Preconditions.checkArgument(children.isEmpty());
+ return new OpenTSDBSubScan(openTSDBStoragePlugin, storage, tabletScanSpecList, columns);
+ }
+
+ @Override
+ public Iterator<PhysicalOperator> iterator() {
+ return Collections.emptyIterator();
+ }
+
+ @Override
+ public <T, X, E extends Throwable> T accept(PhysicalVisitor<T, X, E> physicalVisitor, X value) throws E {
+ return physicalVisitor.visitSubScan(this, value);
+ }
+
+ public List<SchemaPath> getColumns() {
+ return columns;
+ }
+
+ public List<OpenTSDBSubScanSpec> getTabletScanSpecList() {
+ return tabletScanSpecList;
+ }
+
+ @JsonIgnore
+ public OpenTSDBStoragePlugin getStorageEngine() {
+ return openTSDBStoragePlugin;
+ }
+
+ @JsonIgnore
+ public OpenTSDBStoragePluginConfig getStorageConfig() {
--- End diff --
I will fix it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568797
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/OpenTSDBTypes.java ---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+/**
+ * Types in openTSDB records,
+ * used for converting openTSDB data to Sql representation
+ */
+public enum OpenTSDBTypes {
+ STRING,
+ DOUBLE,
+ TIMESTAMP,
--- End diff --
Ok.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957514
--- Diff: contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestOpenTSDBPlugin.java ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.apache.drill.PlanTestBase;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson;
+import static com.github.tomakehurst.wiremock.client.WireMock.get;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;
+import static org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WTIHOUT_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITHOUT_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITH_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.REQUEST_TO_NONEXISTENT_METRIC;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_NAME_REQUEST;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_REQUEST;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITHOUT_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_REQUEST_WITH_TAGS;
+
+public class TestOpenTSDBPlugin extends PlanTestBase {
+
+ protected static OpenTSDBStoragePlugin storagePlugin;
+ protected static OpenTSDBStoragePluginConfig storagePluginConfig;
+
+ @Rule
+ public WireMockRule wireMockRule = new WireMockRule(10000);
+
+ @BeforeClass
+ public static void addTestDataToDB() throws Exception {
--- End diff --
Ok
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567658
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List<SchemaPath> columns;
+ private List<OpenTSDBWork> openTSDBWorkList = Lists.newArrayList();
+ private List<EndpointAffinity> affinities;
+
+ private boolean filterPushedDown = false;
+
+ @JsonCreator
+ public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec") OpenTSDBScanSpec openTSDBScanSpec,
+ @JsonProperty("storage") OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+ @JsonProperty("columns") List<SchemaPath> columns,
--- End diff --
Thank for this advice, I will add getters.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146133808
--- Diff: contrib/storage-opentsdb/src/main/resources/bootstrap-storage-plugins.json ---
@@ -0,0 +1,9 @@
+{
+ "storage": {
+ openTSDB: {
+ type: "openTSDB",
+ connection: "1.2.3.4:4242",
+ enabled: true
--- End diff --
1. Can user have secure connection?
2. I guess it would be better if by default this storage plugin will be disabled.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147582817
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+ private final DrillbitContext context;
+
+ private final OpenTSDBStoragePluginConfig engineConfig;
+ private final OpenTSDBSchemaFactory schemaFactory;
+
+ private final ServiceImpl db;
+
+ public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration, DrillbitContext context, String name) throws IOException {
+ this.context = context;
+ this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+ this.engineConfig = configuration;
+ this.db = new ServiceImpl("http://" + configuration.getConnection());
+ }
+
+ @Override
+ public void start() throws IOException {
--- End diff --
There is no need to override `start` and `close` methods since they have the same behavior as parent...
---
[GitHub] drill issue #774: DRILL-5337: OpenTSDB storage plugin
Posted by dmitriyHavrilovich <gi...@git.apache.org>.
Github user dmitriyHavrilovich commented on the issue:
https://github.com/apache/drill/pull/774
@arina-ielchiieva we done major refactor of code to follow your guideline
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146133164
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Constants.java ---
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+public class Constants {
--- End diff --
interface?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148628583
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/OpenTSDB.java ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import retrofit2.Call;
+import retrofit2.http.Body;
+import retrofit2.http.GET;
+import retrofit2.http.POST;
+import retrofit2.http.Query;
+
+import java.util.Set;
+
+/**
+ * Client for API requests to openTSDB
+ */
+public interface OpenTSDB {
+
+ /**
+ * Used for getting all metrics names from openTSDB
+ *
+ * @return Set<String> with all tables names
+ */
+ @GET("api/suggest?type=metrics&max=999")
--- End diff --
I do not find another way of getting all the metrics from openTSDB. And I don't know in what place we can configure this value. I suppose we can set the max value for this parameter and get all the metrics.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567620
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List<SchemaPath> columns;
+ private List<OpenTSDBWork> openTSDBWorkList = Lists.newArrayList();
+ private List<EndpointAffinity> affinities;
+
+ private boolean filterPushedDown = false;
--- End diff --
No, this variable is not required, I will delete it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148959786
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/Schema.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.*;
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATED_VALUE;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATE_TAGS;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.METRIC;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.TIMESTAMP;
+
+/**
+ * Abstraction for representing structure of openTSDB table
+ */
+public class Schema {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(Schema.class);
+
+ private final List<ColumnDTO> columns = new ArrayList<>();
+ private final Service db;
+ private final String name;
+
+ public Schema(Service db, String name) {
+ this.db = db;
+ this.name = name;
+ setupStructure();
+ }
+
+ private void setupStructure() {
+ columns.add(new ColumnDTO(METRIC.toString(), OpenTSDBTypes.STRING));
+ columns.add(new ColumnDTO(AGGREGATE_TAGS.toString(), OpenTSDBTypes.STRING));
+ columns.add(new ColumnDTO(TIMESTAMP.toString(), OpenTSDBTypes.TIMESTAMP));
+ columns.add(new ColumnDTO(AGGREGATED_VALUE.toString(), OpenTSDBTypes.DOUBLE));
+ columns.addAll(db.getUnfixedColumns(getParamsForQuery()));
+ }
+
+ /**
+ * Return list with all columns names and its types
+ *
+ * @return List<ColumnDTO>
+ */
+ public List<ColumnDTO> getColumns() {
+ return Collections.unmodifiableList(columns);
+ }
+
+ /**
+ * Number of columns in table
+ *
+ * @return number of table columns
+ */
+ public int getColumnCount() {
+ return columns.size();
+ }
+
+ /**
+ * @param columnIndex index of required column in table
+ * @return ColumnDTO
+ */
+ public ColumnDTO getColumnByIndex(int columnIndex) {
+ return columns.get(columnIndex);
+ }
+
+ // Create map with required params, for querying metrics.
+ // Without this params, we cannot make API request to db.
+ private HashMap<String, String> getParamsForQuery() {
+ HashMap<String, String> params = new HashMap<>();
+ params.put(METRIC_PARAM, getValidTableName(name));
+ params.put(AGGREGATOR_PARAM, SUM_AGGREGATOR);
+ params.put(TIME_PARAM, DEFAULT_TIME);
--- End diff --
Please add comment in the code explaining that for this case we need the defaults.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/774
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148934096
--- Diff: contrib/storage-opentsdb/README.md ---
@@ -0,0 +1,64 @@
+# drill-storage-openTSDB
+
+Implementation of TSDB storage plugin. Plugin uses REST API to work with TSDB.
+
+For more information about openTSDB follow this link <http://opentsdb.net>
+
+There is list of required params:
+
+* metric - The name of a metric stored in the db.
+
+* start - The start time for the query. This can be a relative or absolute timestamp.
+
+* aggregator - The name of an aggregation function to use.
+
+optional param is:
+
+* downsample - An optional downsampling function to reduce the amount of data returned.
+
+* end - An end time for the query. If not supplied, the TSD will assume the local system time on the server.
+This may be a relative or absolute timestamp. This param is optional, and if it isn't specified we will send null
+to the db in this field, but in this case db will assume the local system time on the server.
+
+List of supported aggregators
+
+<http://opentsdb.net/docs/build/html/user_guide/query/aggregators.html>
+
+List of supported time
+
+<http://opentsdb.net/docs/build/html/user_guide/query/dates.html>
+
+Params must be specified in FROM clause of the query separated by commas. For example
+
+`openTSDB.(metric=metric_name, start=4d-ago, aggregator=sum)`
+
+Supported queries for now are listed below:
+
+```
+USE openTSDB
+```
+
+```
+SHOW tables
+```
+Will print available metrics. Max number of the printed results is a Integer.MAX value
+
+```
+SELECT * FROM openTSDB. `(metric=warp.speed.test, start=47y-ago, aggregator=sum)`
+```
+Return aggregated elements from `warp.speed.test` table since 47y-ago
+
+```
+SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg, start=47y-ago)`
+```
+Return aggregated elements from `warp.speed.test` table
+
+```
+SELECT `timestamp`, sum(`aggregated value`) FROM openTSDB.`(metric=warp.speed.test, aggregator=avg, start=47y-ago)` GROUP BY `timestamp`
--- End diff --
Please add example with end time + unit test.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569297
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
--- End diff --
Ok, good idea.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569123
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
+ return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set<String> getAllTableNames() {
+ return getTableNames();
+ }
+
+ @Override
+ public List<ColumnDTO> getUnfixedColumnsToSchema() {
+ Set<MetricDTO> tables = getAllMetricsByTags();
+ List<ColumnDTO> unfixedColumns = new ArrayList<>();
+
+ for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+ if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+ }
+ }
+ }
+ return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+ if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+ } else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+ }
+ }
+
+ private Set<MetricDTO> getAllMetricsByTags() {
+ try {
+ return getAllMetricsFromDBByTags();
+ } catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<String> getTableNames() {
+ try {
+ return client.getAllTablesName().execute().body();
+ } catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<MetricDTO> getAllTablesWithSpecialTag(DBQuery base) throws IOException {
+ return client.getTables(base).execute().body();
+ }
+
+ private Set<MetricDTO> getAllMetricsFromDBByTags() throws IOException {
+ Map<String, String> tags = new HashMap<>();
+
+ Query subQuery = new Query.Builder(queryParameters.get(METRIC))
+ .setAggregator(queryParameters.get(AGGREGATOR))
+ .setDownsample(queryParameters.get(DOWNSAMPLE))
+ .setTags(tags).build();
+
+ Set<Query> queries = new HashSet<>();
+ queries.add(subQuery);
+
+ DBQuery baseQuery = new DBQuery.Builder()
+ .setQueries(queries)
+ .build();
+
+ Set<MetricDTO> tables =
+ getBaseTables(baseQuery);
+
+ Set<String> extractedTags =
+ getTagsFromTables(tables);
+
+ tables.clear();
+
+ for (String value : extractedTags) {
+ transformTagsForRequest(tags, value);
--- End diff --
Thank, I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569460
--- Diff: contrib/storage-opentsdb/src/main/resources/bootstrap-storage-plugins.json ---
@@ -0,0 +1,9 @@
+{
+ "storage": {
+ openTSDB: {
+ type: "openTSDB",
+ connection: "1.2.3.4:4242",
+ enabled: true
--- End diff --
For now, a user cannot choose what type of connection will be used by drill. But it must be configurable. I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148934194
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Constants.java ---
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+public interface Constants {
+ /**
+ * openTSDB required constants for API call
+ */
+ public static final String DEFAULT_TIME = "47y-ago";
--- End diff --
Since we agreed that we won't allow defaults, why do we still have these two constants? I also have noticed they are used in the code below...
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569335
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+ private final Map<String, OpenTSDBDatabaseSchema> schemaMap = Maps.newHashMap();
+
+ OpenTSDBTables(String name) {
+ super(Collections.<String>emptyList(), name);
+ }
+
+ @Override
+ public AbstractSchema getSubSchema(String name) {
+ Set<String> tables;
+ if (!schemaMap.containsKey(name)) {
+ tables = plugin.getClient().getAllTableNames();
+ schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, name));
+ }
+ return schemaMap.get(name);
+ }
+
+ @Override
+ public Set<String> getSubSchemaNames() {
+ return Collections.emptySet();
--- End diff --
Yes, I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568996
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
+ return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set<String> getAllTableNames() {
+ return getTableNames();
+ }
+
+ @Override
+ public List<ColumnDTO> getUnfixedColumnsToSchema() {
+ Set<MetricDTO> tables = getAllMetricsByTags();
+ List<ColumnDTO> unfixedColumns = new ArrayList<>();
+
+ for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+ if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+ }
+ }
+ }
+ return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+ if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+ } else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+ }
+ }
+
+ private Set<MetricDTO> getAllMetricsByTags() {
+ try {
+ return getAllMetricsFromDBByTags();
+ } catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<String> getTableNames() {
+ try {
+ return client.getAllTablesName().execute().body();
+ } catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
--- End diff --
This is really bad practice, sorry for that, I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568893
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/Query.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.SUM_AGGREGATOR;
+
+/**
+ * Query is an abstraction of openTSDB subQuery
+ * and it is integral part of DBQuery
+ * <p>
+ * Each sub query can retrieve individual or groups of timeseries data,
+ * performing aggregation on each set.
+ */
+public class Query {
+
+ /**
+ * The name of an aggregation function to use.
+ */
+ private String aggregator;
+ /**
+ * The name of a metric stored in the system
+ */
+ private String metric;
+ /**
+ * Whether or not the data should be converted into deltas before returning.
+ * This is useful if the metric is a continuously incrementing counter
+ * and you want to view the rate of change between data points.
+ */
+ private String rate;
+ /**
+ * An optional downsampling function to reduce the amount of data returned.
+ */
+ private String downsample;
+ /**
+ * To drill down to specific timeseries or group results by tag,
+ * supply one or more map values in the same format as the query string.
+ */
+ private Map<String, String> tags;
+
+ private Query(Builder builder) {
+ this.aggregator = builder.aggregator;
+ this.metric = builder.metric;
+ this.rate = builder.rate;
+ this.downsample = builder.downsample;
+ this.tags = builder.tags;
+ }
+
+ public String getAggregator() {
+ return aggregator;
+ }
+
+ public String getMetric() {
+ return metric;
+ }
+
+ public String getRate() {
+ return rate;
+ }
+
+ public String getDownsample() {
+ return downsample;
+ }
+
+ public Map<String, String> getTags() {
+ return tags;
+ }
+
+ public static class Builder {
+
+ private String aggregator = SUM_AGGREGATOR;
+ private String metric;
+ private String rate;
+ private String downsample;
+ private Map<String, String> tags = new HashMap<>();
+
+ public Builder(String metric) {
+ this.metric = metric;
+ }
+
+ public Builder setAggregator(String aggregator) {
+ if (aggregator != null) {
+ this.aggregator = aggregator;
+ }
+ return this;
+ }
+
+ public Builder setMetric(String metric) {
+ this.metric = metric;
+ return this;
+ }
+
+ public Builder setRate(String rate) {
+ if (rate != null) {
--- End diff --
Ok, I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568766
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Util.java ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.base.Splitter;
+
+import java.util.Map;
+
+public class Util {
+
+ /**
+ * Parse FROM parameters to Map representation
+ *
+ * @param rowData with this syntax (metric=warp.speed.test)
+ * @return Map with params key: metric, value: warp.speed.test
+ */
+ public static Map<String, String> parseFROMRowData(String rowData) {
--- End diff --
I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148933988
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/Schema.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.*;
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATED_VALUE;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATE_TAGS;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.METRIC;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.TIMESTAMP;
+
+/**
+ * Abstraction for representing structure of openTSDB table
+ */
+public class Schema {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(Schema.class);
+
+ private final List<ColumnDTO> columns = new ArrayList<>();
+ private final Service db;
+ private final String name;
+
+ public Schema(Service db, String name) {
+ this.db = db;
+ this.name = name;
+ setupStructure();
+ }
+
+ private void setupStructure() {
+ columns.add(new ColumnDTO(METRIC.toString(), OpenTSDBTypes.STRING));
+ columns.add(new ColumnDTO(AGGREGATE_TAGS.toString(), OpenTSDBTypes.STRING));
+ columns.add(new ColumnDTO(TIMESTAMP.toString(), OpenTSDBTypes.TIMESTAMP));
+ columns.add(new ColumnDTO(AGGREGATED_VALUE.toString(), OpenTSDBTypes.DOUBLE));
+ columns.addAll(db.getUnfixedColumns(getParamsForQuery()));
+ }
+
+ /**
+ * Return list with all columns names and its types
+ *
+ * @return List<ColumnDTO>
+ */
+ public List<ColumnDTO> getColumns() {
+ return Collections.unmodifiableList(columns);
+ }
+
+ /**
+ * Number of columns in table
+ *
+ * @return number of table columns
+ */
+ public int getColumnCount() {
+ return columns.size();
+ }
+
+ /**
+ * @param columnIndex index of required column in table
+ * @return ColumnDTO
+ */
+ public ColumnDTO getColumnByIndex(int columnIndex) {
+ return columns.get(columnIndex);
+ }
+
+ // Create map with required params, for querying metrics.
+ // Without this params, we cannot make API request to db.
+ private HashMap<String, String> getParamsForQuery() {
--- End diff --
`Map`
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567893
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
+ List<SchemaPath> projectedColumns) throws IOException {
+ setColumns(projectedColumns);
+ this.db = client;
+ this.subScanSpec = subScanSpec;
+ db.setupQueryParameters(subScanSpec.getTableName());
+ log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws ExecutionSetupException {
+ this.output = output;
+ Set<MetricDTO> tables = db.getTablesFromDB();
+ if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or it's empty", subScanSpec.getTableName()));
+ }
+ this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+ try {
+ return processOpenTSDBTablesData();
+ } catch (SchemaChangeException e) {
+ log.info(e.toString());
+ return 0;
+ }
+ }
+
+ @Override
+ protected boolean isSkipQuery() {
+ return super.isSkipQuery();
+ }
+
+ @Override
+ public void close() throws Exception {
+ }
+
+ static {
+ TYPES = ImmutableMap.<OpenTSDBTypes, MinorType>builder()
+ .put(OpenTSDBTypes.STRING, MinorType.VARCHAR)
+ .put(OpenTSDBTypes.DOUBLE, MinorType.FLOAT8)
+ .put(OpenTSDBTypes.TIMESTAMP, MinorType.TIMESTAMP)
+ .build();
+ }
+
+ private static class ProjectedColumnInfo {
+ ValueVector vv;
+ ColumnDTO openTSDBColumn;
+ }
+
+ private int processOpenTSDBTablesData() throws SchemaChangeException {
+ int rowCounter = 0;
+ while (tableIterator.hasNext()) {
+ MetricDTO metricDTO = tableIterator.next();
+ rowCounter = addRowResult(metricDTO, rowCounter);
+ }
+ return rowCounter;
+ }
+
+ private int addRowResult(MetricDTO table, int rowCounter) throws SchemaChangeException {
+ setupProjectedColsIfItNull();
+ for (String time : table.getDps().keySet()) {
+ String value = table.getDps().get(time);
+ setupDataToDrillTable(table, time, value, table.getTags(), rowCounter);
+ rowCounter++;
+ }
+ return rowCounter;
+ }
+
+ private void setupProjectedColsIfItNull() throws SchemaChangeException {
+ if (projectedCols == null) {
+ initCols(new Schema(db, subScanSpec.getTableName()));
+ }
+ }
+
+ private void setupDataToDrillTable(MetricDTO table, String timestamp, String value, Map<String, String> tags, int rowCount) {
+ for (ProjectedColumnInfo pci : projectedCols) {
+ switch (pci.openTSDBColumn.getColumnName()) {
+ case "metric":
+ setStringColumnValue(table.getMetric(), pci, rowCount);
+ break;
+ case "aggregate tags":
+ setStringColumnValue(table.getAggregateTags().toString(), pci, rowCount);
+ break;
+ case "timestamp":
+ setTimestampColumnValue(timestamp, pci, rowCount);
+ break;
+ case "aggregated value":
+ setDoubleColumnValue(value, pci, rowCount);
+ break;
+ default:
+ setStringColumnValue(tags.get(pci.openTSDBColumn.getColumnName()), pci, rowCount);
+ }
+ }
+ }
+
+ private void setTimestampColumnValue(String timestamp, ProjectedColumnInfo pci, int rowCount) {
+ setTimestampColumnValue(timestamp != null ? Long.parseLong(timestamp) : Long.parseLong("0"), pci, rowCount);
+ }
+
+ private void setDoubleColumnValue(String value, ProjectedColumnInfo pci, int rowCount) {
+ setDoubleColumnValue(value != null ? Double.parseDouble(value) : 0.0, pci, rowCount);
+ }
+
+ private void setStringColumnValue(String data, ProjectedColumnInfo pci, int rowCount) {
+ if (data == null) {
+ data = "null";
+ }
+ ByteBuffer value = ByteBuffer.wrap(data.getBytes(UTF_8));
+ ((NullableVarCharVector.Mutator) pci.vv.getMutator())
+ .setSafe(rowCount, value, 0, value.remaining());
+ }
+
+ private void setTimestampColumnValue(Long data, ProjectedColumnInfo pci, int rowCount) {
+ ((NullableTimeStampVector.Mutator) pci.vv.getMutator())
+ .setSafe(rowCount, data * 1000);
+ }
+
+ private void setDoubleColumnValue(Double data, ProjectedColumnInfo pci, int rowCount) {
+ ((NullableFloat8Vector.Mutator) pci.vv.getMutator())
+ .setSafe(rowCount, data);
+ }
+
+ private void initCols(Schema schema) throws SchemaChangeException {
+ ImmutableList.Builder<ProjectedColumnInfo> pciBuilder = ImmutableList.builder();
+
+ for (int i = 0; i < schema.getColumnCount(); i++) {
+
+ ColumnDTO column = schema.getColumnByIndex(i);
+ final String name = column.getColumnName();
+ final OpenTSDBTypes type = column.getColumnType();
+ TypeProtos.MinorType minorType = TYPES.get(type);
+
+ if (isMinorTypeNull(minorType)) {
+ logExceptionMessage(name, type);
--- End diff --
Ok, I will add an exception with a detailed message.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148934296
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
--- End diff --
`openTSDB-sub-scan`
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147583640
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
+ return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set<String> getAllTableNames() {
+ return getTableNames();
+ }
+
+ @Override
+ public List<ColumnDTO> getUnfixedColumnsToSchema() {
+ Set<MetricDTO> tables = getAllMetricsByTags();
+ List<ColumnDTO> unfixedColumns = new ArrayList<>();
+
+ for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+ if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+ }
+ }
+ }
+ return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+ if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+ } else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+ }
+ }
+
+ private Set<MetricDTO> getAllMetricsByTags() {
+ try {
+ return getAllMetricsFromDBByTags();
+ } catch (IOException e) {
+ logIOException(e);
--- End diff --
Why do we log error rather than fail? Is failure non critical?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147583241
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Util.java ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.base.Splitter;
+
+import java.util.Map;
+
+public class Util {
+
+ /**
+ * Parse FROM parameters to Map representation
+ *
+ * @param rowData with this syntax (metric=warp.speed.test)
+ * @return Map with params key: metric, value: warp.speed.test
+ */
+ public static Map<String, String> parseFROMRowData(String rowData) {
--- End diff --
`parseFromRowData`
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by vladimirtkach <gi...@git.apache.org>.
Github user vladimirtkach commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r143679664
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePluginConfig.java ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.drill.common.logical.StoragePluginConfigBase;
+
+import java.io.IOException;
+import java.util.Objects;
+
+@JsonTypeName(OpenTSDBStoragePluginConfig.NAME)
+public class OpenTSDBStoragePluginConfig extends StoragePluginConfigBase {
+
+ public static final String NAME = "openTSDB";
+
+ private final String connection;
+
+ @JsonCreator
+ public OpenTSDBStoragePluginConfig(@JsonProperty("connection") String connection) throws IOException {
+ this.connection = connection;
+ }
+
+ String getConnection() {
--- End diff --
I have noticed that there is an error during plugin configuration serialization, this happens because method accessor is not public.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569425
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+ private final Map<String, OpenTSDBDatabaseSchema> schemaMap = Maps.newHashMap();
+
+ OpenTSDBTables(String name) {
+ super(Collections.<String>emptyList(), name);
+ }
+
+ @Override
+ public AbstractSchema getSubSchema(String name) {
+ Set<String> tables;
+ if (!schemaMap.containsKey(name)) {
+ tables = plugin.getClient().getAllTableNames();
+ schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, name));
+ }
+ return schemaMap.get(name);
+ }
+
+ @Override
+ public Set<String> getSubSchemaNames() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+ return new DrillOpenTSDBTable(schemaName, plugin, new Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+ log.warn("Failure while retrieving openTSDB table {}", name, e);
+ return null;
+ }
+ }
+
+ @Override
+ public Set<String> getTableNames() {
+ return plugin.getClient().getAllTableNames();
+ }
+
+ @Override
+ public CreateTableEntry createNewTable(final String tableName, List<String> partitionColumns) {
+ return null;
+ }
+
+ @Override
+ public void dropTable(String tableName) {
+ }
+
+ @Override
+ public boolean isMutable() {
--- End diff --
Yes, it is a mistake, I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146133068
--- Diff: distribution/src/assemble/bin.xml ---
@@ -99,7 +99,9 @@
<include>org.apache.drill.contrib:drill-format-mapr</include>
<include>org.apache.drill.contrib:drill-jdbc-storage</include>
<include>org.apache.drill.contrib:drill-kudu-storage</include>
+ <include>org.apache.drill.contrib:drill-opentsdb-storage</include>
--- End diff --
Again we have `<include>org.apache.drill.contrib:drill-opentsdb-storage</include>` twice in here...
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147582484
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/OpenTSDBTypes.java ---
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+/**
+ * Types in openTSDB records,
+ * used for converting openTSDB data to Sql representation
+ */
+public enum OpenTSDBTypes {
+ STRING,
+ DOUBLE,
+ TIMESTAMP,
--- End diff --
Comma can be removed.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146133289
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/OpenTSDB.java ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import retrofit2.Call;
+import retrofit2.http.Body;
+import retrofit2.http.GET;
+import retrofit2.http.POST;
+import retrofit2.http.Query;
+
+import java.util.Set;
+
+/**
+ * Client for API requests to openTSDB
+ */
+public interface OpenTSDB {
+
+ /**
+ * Used for getting all metrics names from openTSDB
+ *
+ * @return Set<String> with all tables names
+ */
+ @GET("api/suggest?type=metrics&max=999")
--- End diff --
Why max value is 999? What is we have more metrics? If it is required can we make it configurable?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146133035
--- Diff: contrib/storage-opentsdb/src/test/resources/logback.xml ---
@@ -0,0 +1,64 @@
+<?xml version="1.0" encoding="UTF-8" ?>
--- End diff --
Now in Drill we have common logback-test.xml for all modules, please use it instead of creating separate one.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957509
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/Schema.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.*;
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATED_VALUE;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATE_TAGS;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.METRIC;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.TIMESTAMP;
+
+/**
+ * Abstraction for representing structure of openTSDB table
+ */
+public class Schema {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(Schema.class);
+
+ private final List<ColumnDTO> columns = new ArrayList<>();
+ private final Service db;
+ private final String name;
+
+ public Schema(Service db, String name) {
+ this.db = db;
+ this.name = name;
+ setupStructure();
+ }
+
+ private void setupStructure() {
+ columns.add(new ColumnDTO(METRIC.toString(), OpenTSDBTypes.STRING));
+ columns.add(new ColumnDTO(AGGREGATE_TAGS.toString(), OpenTSDBTypes.STRING));
+ columns.add(new ColumnDTO(TIMESTAMP.toString(), OpenTSDBTypes.TIMESTAMP));
+ columns.add(new ColumnDTO(AGGREGATED_VALUE.toString(), OpenTSDBTypes.DOUBLE));
+ columns.addAll(db.getUnfixedColumns(getParamsForQuery()));
+ }
+
+ /**
+ * Return list with all columns names and its types
+ *
+ * @return List<ColumnDTO>
+ */
+ public List<ColumnDTO> getColumns() {
+ return Collections.unmodifiableList(columns);
+ }
+
+ /**
+ * Number of columns in table
+ *
+ * @return number of table columns
+ */
+ public int getColumnCount() {
+ return columns.size();
+ }
+
+ /**
+ * @param columnIndex index of required column in table
+ * @return ColumnDTO
+ */
+ public ColumnDTO getColumnByIndex(int columnIndex) {
+ return columns.get(columnIndex);
+ }
+
+ // Create map with required params, for querying metrics.
+ // Without this params, we cannot make API request to db.
+ private HashMap<String, String> getParamsForQuery() {
+ HashMap<String, String> params = new HashMap<>();
+ params.put(METRIC_PARAM, getValidTableName(name));
+ params.put(AGGREGATOR_PARAM, SUM_AGGREGATOR);
+ params.put(TIME_PARAM, DEFAULT_TIME);
--- End diff --
Explained this in the comment above.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146134207
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
+ return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set<String> getAllTableNames() {
+ return getTableNames();
+ }
+
+ @Override
+ public List<ColumnDTO> getUnfixedColumnsToSchema() {
+ Set<MetricDTO> tables = getAllMetricsByTags();
+ List<ColumnDTO> unfixedColumns = new ArrayList<>();
+
+ for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+ if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+ }
+ }
+ }
+ return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+ if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+ } else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+ }
+ }
+
+ private Set<MetricDTO> getAllMetricsByTags() {
+ try {
+ return getAllMetricsFromDBByTags();
+ } catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<String> getTableNames() {
+ try {
+ return client.getAllTablesName().execute().body();
+ } catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<MetricDTO> getAllTablesWithSpecialTag(DBQuery base) throws IOException {
+ return client.getTables(base).execute().body();
+ }
+
+ private Set<MetricDTO> getAllMetricsFromDBByTags() throws IOException {
+ Map<String, String> tags = new HashMap<>();
+
+ Query subQuery = new Query.Builder(queryParameters.get(METRIC))
+ .setAggregator(queryParameters.get(AGGREGATOR))
+ .setDownsample(queryParameters.get(DOWNSAMPLE))
+ .setTags(tags).build();
+
+ Set<Query> queries = new HashSet<>();
+ queries.add(subQuery);
+
+ DBQuery baseQuery = new DBQuery.Builder()
+ .setQueries(queries)
+ .build();
+
+ Set<MetricDTO> tables =
+ getBaseTables(baseQuery);
+
+ Set<String> extractedTags =
+ getTagsFromTables(tables);
+
+ tables.clear();
+
+ for (String value : extractedTags) {
+ transformTagsForRequest(tags, value);
+ tables.addAll(getAllTablesWithSpecialTag(baseQuery));
+ }
+ return tables;
+ }
+
+ private Set<MetricDTO> getBaseTables(DBQuery base) throws IOException {
+ return getAllTablesWithSpecialTag(base);
+ }
+
+ private Set<String> getTagsFromTables(Set<MetricDTO> tables) {
+ Set<String> extractedTags = new HashSet<>();
+
+ for (MetricDTO table : tables) {
+ extractedTags.addAll(table.getAggregateTags());
+ extractedTags.addAll(table.getTags().keySet());
+ }
+
+ return extractedTags;
+ }
+
+ private void transformTagsForRequest(Map<String, String> tags, String value) {
+ tags.clear();
+ tags.put(value, "*");
+ }
+
+ private void logIOException(IOException e) {
+ log.warn("A problem occurred when talking to the server", e);
--- End diff --
That definitely should not be a warn and rather returning meaningful error to the user.
---
[GitHub] drill issue #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/774
@Vlad-Storona I have deployed your branch and have noticed several more issues:
1. When I openTSDB storage plugin tab for the first time, its content is showing as null.
2. When I have indicated incorrect connection, the following exception is displayed in the log:
```
org.apache.drill.common.exceptions.ExecutionSetupException: Failure setting up new storage plugin configuration for config org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig@a837d5e8
```
3. When I try to query with incorrect syntax, the following error is displayed:
```
0: jdbc:drill:drillbit=localhost> select * from `mymetric.stock`;
Error: SYSTEM ERROR: IllegalArgumentException: Chunk [mymetric.stock] is not a valid entry
Caused by: java.lang.IllegalArgumentException: Chunk [mymetric.stock] is not a valid entry
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) ~[guava-18.0.jar:na]
at com.google.common.base.Splitter$MapSplitter.split(Splitter.java:508) ~[guava-18.0.jar:na]
```
4. When I try to enable storage plugin config with incorrect connection, it fails. I think we should allow enabling config even if we could not connect to openTSDB as others plugins do but fail when we try to query the data.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568092
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
+public class OpenTSDBSubScan extends AbstractBase implements SubScan {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(OpenTSDBSubScan.class);
+
+ @JsonProperty
+ public final OpenTSDBStoragePluginConfig storage;
+
+ private final List<SchemaPath> columns;
+ private final OpenTSDBStoragePlugin openTSDBStoragePlugin;
+ private final List<OpenTSDBSubScanSpec> tabletScanSpecList;
+
+ @JsonCreator
+ public OpenTSDBSubScan(@JacksonInject StoragePluginRegistry registry,
+ @JsonProperty("storage") StoragePluginConfig storage,
+ @JsonProperty("tabletScanSpecList") LinkedList<OpenTSDBSubScanSpec> tabletScanSpecList,
+ @JsonProperty("columns") List<SchemaPath> columns) throws ExecutionSetupException {
+ super((String) null);
+ openTSDBStoragePlugin = (OpenTSDBStoragePlugin) registry.getPlugin(storage);
+ this.tabletScanSpecList = tabletScanSpecList;
+ this.storage = (OpenTSDBStoragePluginConfig) storage;
--- End diff --
Thanks. I will change this casting.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148934383
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/Schema.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client;
+
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.*;
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATED_VALUE;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.AGGREGATE_TAGS;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.METRIC;
+import static org.apache.drill.exec.store.openTSDB.client.Schema.DefaultColumns.TIMESTAMP;
+
+/**
+ * Abstraction for representing structure of openTSDB table
+ */
+public class Schema {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(Schema.class);
+
+ private final List<ColumnDTO> columns = new ArrayList<>();
+ private final Service db;
+ private final String name;
+
+ public Schema(Service db, String name) {
+ this.db = db;
+ this.name = name;
+ setupStructure();
+ }
+
+ private void setupStructure() {
+ columns.add(new ColumnDTO(METRIC.toString(), OpenTSDBTypes.STRING));
+ columns.add(new ColumnDTO(AGGREGATE_TAGS.toString(), OpenTSDBTypes.STRING));
+ columns.add(new ColumnDTO(TIMESTAMP.toString(), OpenTSDBTypes.TIMESTAMP));
+ columns.add(new ColumnDTO(AGGREGATED_VALUE.toString(), OpenTSDBTypes.DOUBLE));
+ columns.addAll(db.getUnfixedColumns(getParamsForQuery()));
+ }
+
+ /**
+ * Return list with all columns names and its types
+ *
+ * @return List<ColumnDTO>
+ */
+ public List<ColumnDTO> getColumns() {
+ return Collections.unmodifiableList(columns);
+ }
+
+ /**
+ * Number of columns in table
+ *
+ * @return number of table columns
+ */
+ public int getColumnCount() {
+ return columns.size();
+ }
+
+ /**
+ * @param columnIndex index of required column in table
+ * @return ColumnDTO
+ */
+ public ColumnDTO getColumnByIndex(int columnIndex) {
+ return columns.get(columnIndex);
+ }
+
+ // Create map with required params, for querying metrics.
+ // Without this params, we cannot make API request to db.
+ private HashMap<String, String> getParamsForQuery() {
+ HashMap<String, String> params = new HashMap<>();
+ params.put(METRIC_PARAM, getValidTableName(name));
+ params.put(AGGREGATOR_PARAM, SUM_AGGREGATOR);
+ params.put(TIME_PARAM, DEFAULT_TIME);
--- End diff --
Why do we use defaults here?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148934449
--- Diff: exec/java-exec/src/test/resources/drill-module.conf ---
@@ -10,7 +10,7 @@ drill: {
packages += "org.apache.drill.exec.testing",
packages += "org.apache.drill.exec.rpc.user.security.testing"
}
- test.query.printing.silent : false,
+ test.query.printing.silent : false,
--- End diff --
Please remove changes in this file!
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146135112
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List<SchemaPath> columns;
+ private List<OpenTSDBWork> openTSDBWorkList = Lists.newArrayList();
+ private List<EndpointAffinity> affinities;
+
+ private boolean filterPushedDown = false;
+
+ @JsonCreator
+ public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec") OpenTSDBScanSpec openTSDBScanSpec,
+ @JsonProperty("storage") OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+ @JsonProperty("columns") List<SchemaPath> columns,
+ @JacksonInject StoragePluginRegistry pluginRegistry) throws IOException, ExecutionSetupException {
+ this((OpenTSDBStoragePlugin) pluginRegistry.getPlugin(openTSDBStoragePluginConfig), openTSDBScanSpec, columns);
+ }
+
+ public OpenTSDBGroupScan(OpenTSDBStoragePlugin storagePlugin,
+ OpenTSDBScanSpec scanSpec, List<SchemaPath> columns) {
+ super((String) null);
+ this.storagePlugin = storagePlugin;
+ this.storagePluginConfig = storagePlugin.getConfig();
+ this.openTSDBScanSpec = scanSpec;
+ this.columns = columns == null || columns.size() == 0 ? ALL_COLUMNS : columns;
+ init();
+ }
+
+ /**
+ * Private constructor, used for cloning.
+ *
+ * @param that The OpenTSDBGroupScan to clone
+ */
+ private OpenTSDBGroupScan(OpenTSDBGroupScan that) {
+ super((String) null);
+ this.columns = that.columns;
+ this.openTSDBScanSpec = that.openTSDBScanSpec;
+ this.storagePlugin = that.storagePlugin;
+ this.storagePluginConfig = that.storagePluginConfig;
+ this.filterPushedDown = that.filterPushedDown;
+ this.openTSDBWorkList = that.openTSDBWorkList;
+ this.assignments = that.assignments;
+ init();
+ }
+
+ private void init() {
+ Collection<DrillbitEndpoint> endpoints = storagePlugin.getContext().getBits();
+ Map<String, DrillbitEndpoint> endpointMap = Maps.newHashMap();
+
+ for (DrillbitEndpoint endpoint : endpoints) {
+ endpointMap.put(endpoint.getAddress(), endpoint);
+ }
+ }
+
+ @Override
+ public List<EndpointAffinity> getOperatorAffinity() {
+ if (affinities == null) {
+ affinities = AffinityCreator.getAffinityMap(openTSDBWorkList);
+ }
+ return affinities;
+ }
+
+ @Override
+ public int getMaxParallelizationWidth() {
+ return openTSDBWorkList.size();
+ }
+
+ @Override
+ public void applyAssignments(List<DrillbitEndpoint> incomingEndpoints) {
+ assignments = AssignmentCreator.getMappings(incomingEndpoints, openTSDBWorkList);
+ }
+
+ @Override
+ public OpenTSDBSubScan getSpecificScan(int minorFragmentId) {
+ List<OpenTSDBSubScanSpec> scanSpecList = Lists.newArrayList();
+ scanSpecList.add(new OpenTSDBSubScanSpec(getTableName()));
+ return new OpenTSDBSubScan(storagePlugin, storagePluginConfig, scanSpecList, this.columns);
+ }
+
+ @Override
+ public ScanStats getScanStats() {
+ //magic number ?
+ long recordCount = 100000L * openTSDBWorkList.size();
--- End diff --
Can you please explain this calculation?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569487
--- Diff: contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestDataHolder.java ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+class TestDataHolder {
--- End diff --
Ok.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568843
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.DEFAULT_TIME;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request to DB.
+ * <p>
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result set.
+ */
+public class DBQuery {
+
+ /**
+ * The start time for the query. This can be a relative or absolute timestamp.
+ */
+ private String start;
+ /**
+ * One or more sub subQueries used to select the time series to return.
+ */
+ private Set<Query> queries;
+
+ private DBQuery(Builder builder) {
+ this.start = builder.start;
+ this.queries = builder.queries;
+ }
+
+ public String getStart() {
+ return start;
+ }
+
+ public Set<Query> getQueries() {
+ return queries;
+ }
+
+ public static class Builder {
+
+ private String start = DEFAULT_TIME;
+ private Set<Query> queries = new HashSet<>();
--- End diff --
Yes, good idea, I will add this type of check to the `Builder`.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568871
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/Query.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.SUM_AGGREGATOR;
+
+/**
+ * Query is an abstraction of openTSDB subQuery
+ * and it is integral part of DBQuery
+ * <p>
+ * Each sub query can retrieve individual or groups of timeseries data,
+ * performing aggregation on each set.
+ */
+public class Query {
+
+ /**
+ * The name of an aggregation function to use.
+ */
+ private String aggregator;
+ /**
+ * The name of a metric stored in the system
+ */
+ private String metric;
+ /**
+ * Whether or not the data should be converted into deltas before returning.
+ * This is useful if the metric is a continuously incrementing counter
+ * and you want to view the rate of change between data points.
+ */
+ private String rate;
+ /**
+ * An optional downsampling function to reduce the amount of data returned.
+ */
+ private String downsample;
+ /**
+ * To drill down to specific timeseries or group results by tag,
+ * supply one or more map values in the same format as the query string.
+ */
+ private Map<String, String> tags;
+
+ private Query(Builder builder) {
+ this.aggregator = builder.aggregator;
+ this.metric = builder.metric;
+ this.rate = builder.rate;
+ this.downsample = builder.downsample;
+ this.tags = builder.tags;
+ }
+
+ public String getAggregator() {
+ return aggregator;
+ }
+
+ public String getMetric() {
+ return metric;
+ }
+
+ public String getRate() {
+ return rate;
+ }
+
+ public String getDownsample() {
+ return downsample;
+ }
+
+ public Map<String, String> getTags() {
+ return tags;
+ }
+
+ public static class Builder {
+
+ private String aggregator = SUM_AGGREGATOR;
--- End diff --
Sure, I will remove support for these type of queries.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569621
--- Diff: distribution/src/assemble/bin.xml ---
@@ -99,7 +99,9 @@
<include>org.apache.drill.contrib:drill-format-mapr</include>
<include>org.apache.drill.contrib:drill-jdbc-storage</include>
<include>org.apache.drill.contrib:drill-kudu-storage</include>
+ <include>org.apache.drill.contrib:drill-opentsdb-storage</include>
--- End diff --
It is the same problem as in pom file. I will solve it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147583562
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/Query.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.SUM_AGGREGATOR;
+
+/**
+ * Query is an abstraction of openTSDB subQuery
+ * and it is integral part of DBQuery
+ * <p>
+ * Each sub query can retrieve individual or groups of timeseries data,
+ * performing aggregation on each set.
+ */
+public class Query {
+
+ /**
+ * The name of an aggregation function to use.
+ */
+ private String aggregator;
+ /**
+ * The name of a metric stored in the system
+ */
+ private String metric;
+ /**
+ * Whether or not the data should be converted into deltas before returning.
+ * This is useful if the metric is a continuously incrementing counter
+ * and you want to view the rate of change between data points.
+ */
+ private String rate;
+ /**
+ * An optional downsampling function to reduce the amount of data returned.
+ */
+ private String downsample;
+ /**
+ * To drill down to specific timeseries or group results by tag,
+ * supply one or more map values in the same format as the query string.
+ */
+ private Map<String, String> tags;
+
+ private Query(Builder builder) {
+ this.aggregator = builder.aggregator;
+ this.metric = builder.metric;
+ this.rate = builder.rate;
+ this.downsample = builder.downsample;
+ this.tags = builder.tags;
+ }
+
+ public String getAggregator() {
+ return aggregator;
+ }
+
+ public String getMetric() {
+ return metric;
+ }
+
+ public String getRate() {
+ return rate;
+ }
+
+ public String getDownsample() {
+ return downsample;
+ }
+
+ public Map<String, String> getTags() {
+ return tags;
+ }
+
+ public static class Builder {
+
+ private String aggregator = SUM_AGGREGATOR;
--- End diff --
Aggregation is required parameter [1], should fail and ask user to provide one or even suggest list of available aggragators rather then using made up default. Taking into account OpenTSDB specifics it's rather confusing allowing the following queries:
```
SELECT * FROM openTSDB.`warp.speed.test`
```
[1] http://opentsdb.net/docs/build/html/api_http/query/index.html
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148628710
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBDatabaseSchema.java ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory.OpenTSDBTables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import java.util.Set;
+
+public class OpenTSDBDatabaseSchema extends AbstractSchema {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(OpenTSDBDatabaseSchema.class);
+
+ private final OpenTSDBTables schema;
+ private final Set<String> tableNames;
+
+ private final Map<String, DrillTable> drillTables = Maps.newHashMap();
+
+ public OpenTSDBDatabaseSchema(Set<String> tableList, OpenTSDBTables schema,
+ String name) {
+ super(schema.getSchemaPath(), name);
+ this.schema = schema;
+ this.tableNames = Sets.newHashSet(tableList);
--- End diff --
I rewrited `OpenTSDBSchemaFactory`, and for that moment `OpenTSDBDatabaseSchema` not needed.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569640
--- Diff: exec/java-exec/src/test/resources/drill-module.conf ---
@@ -10,7 +10,7 @@ drill: {
packages += "org.apache.drill.exec.testing",
packages += "org.apache.drill.exec.rpc.user.security.testing"
}
- test.query.printing.silent : false,
+ test.query.printing.silent : false,
--- End diff --
Ok.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568822
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.DEFAULT_TIME;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request to DB.
+ * <p>
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result set.
+ */
+public class DBQuery {
+
+ /**
+ * The start time for the query. This can be a relative or absolute timestamp.
+ */
+ private String start;
+ /**
+ * One or more sub subQueries used to select the time series to return.
+ */
+ private Set<Query> queries;
+
+ private DBQuery(Builder builder) {
+ this.start = builder.start;
+ this.queries = builder.queries;
+ }
+
+ public String getStart() {
+ return start;
+ }
+
+ public Set<Query> getQueries() {
+ return queries;
+ }
+
+ public static class Builder {
+
+ private String start = DEFAULT_TIME;
--- End diff --
Sure, I will change this behavior, and this parameter will be required for all select queries.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567870
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
+ List<SchemaPath> projectedColumns) throws IOException {
+ setColumns(projectedColumns);
+ this.db = client;
+ this.subScanSpec = subScanSpec;
+ db.setupQueryParameters(subScanSpec.getTableName());
+ log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws ExecutionSetupException {
+ this.output = output;
+ Set<MetricDTO> tables = db.getTablesFromDB();
+ if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or it's empty", subScanSpec.getTableName()));
+ }
+ this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+ try {
+ return processOpenTSDBTablesData();
+ } catch (SchemaChangeException e) {
+ log.info(e.toString());
+ return 0;
+ }
+ }
+
+ @Override
+ protected boolean isSkipQuery() {
+ return super.isSkipQuery();
+ }
+
+ @Override
+ public void close() throws Exception {
+ }
+
+ static {
+ TYPES = ImmutableMap.<OpenTSDBTypes, MinorType>builder()
+ .put(OpenTSDBTypes.STRING, MinorType.VARCHAR)
+ .put(OpenTSDBTypes.DOUBLE, MinorType.FLOAT8)
+ .put(OpenTSDBTypes.TIMESTAMP, MinorType.TIMESTAMP)
+ .build();
+ }
+
+ private static class ProjectedColumnInfo {
+ ValueVector vv;
+ ColumnDTO openTSDBColumn;
+ }
+
+ private int processOpenTSDBTablesData() throws SchemaChangeException {
+ int rowCounter = 0;
+ while (tableIterator.hasNext()) {
+ MetricDTO metricDTO = tableIterator.next();
+ rowCounter = addRowResult(metricDTO, rowCounter);
+ }
+ return rowCounter;
+ }
+
+ private int addRowResult(MetricDTO table, int rowCounter) throws SchemaChangeException {
+ setupProjectedColsIfItNull();
+ for (String time : table.getDps().keySet()) {
+ String value = table.getDps().get(time);
+ setupDataToDrillTable(table, time, value, table.getTags(), rowCounter);
+ rowCounter++;
+ }
+ return rowCounter;
+ }
+
+ private void setupProjectedColsIfItNull() throws SchemaChangeException {
+ if (projectedCols == null) {
+ initCols(new Schema(db, subScanSpec.getTableName()));
+ }
+ }
+
+ private void setupDataToDrillTable(MetricDTO table, String timestamp, String value, Map<String, String> tags, int rowCount) {
+ for (ProjectedColumnInfo pci : projectedCols) {
+ switch (pci.openTSDBColumn.getColumnName()) {
+ case "metric":
+ setStringColumnValue(table.getMetric(), pci, rowCount);
+ break;
+ case "aggregate tags":
+ setStringColumnValue(table.getAggregateTags().toString(), pci, rowCount);
+ break;
+ case "timestamp":
+ setTimestampColumnValue(timestamp, pci, rowCount);
+ break;
+ case "aggregated value":
+ setDoubleColumnValue(value, pci, rowCount);
+ break;
+ default:
+ setStringColumnValue(tags.get(pci.openTSDBColumn.getColumnName()), pci, rowCount);
+ }
+ }
+ }
+
+ private void setTimestampColumnValue(String timestamp, ProjectedColumnInfo pci, int rowCount) {
+ setTimestampColumnValue(timestamp != null ? Long.parseLong(timestamp) : Long.parseLong("0"), pci, rowCount);
--- End diff --
I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147583342
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.DEFAULT_TIME;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request to DB.
+ * <p>
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result set.
+ */
+public class DBQuery {
+
+ /**
+ * The start time for the query. This can be a relative or absolute timestamp.
+ */
+ private String start;
+ /**
+ * One or more sub subQueries used to select the time series to return.
+ */
+ private Set<Query> queries;
+
+ private DBQuery(Builder builder) {
+ this.start = builder.start;
+ this.queries = builder.queries;
+ }
+
+ public String getStart() {
+ return start;
+ }
+
+ public Set<Query> getQueries() {
+ return queries;
+ }
+
+ public static class Builder {
+
+ private String start = DEFAULT_TIME;
--- End diff --
According to the documentation start time is required parameter, I we should not use our made up default but rather fail and ask user to provide start time.
[1] http://opentsdb.net/docs/build/html/api_http/query/index.html
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146135302
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
+ List<SchemaPath> projectedColumns) throws IOException {
+ setColumns(projectedColumns);
+ this.db = client;
+ this.subScanSpec = subScanSpec;
+ db.setupQueryParameters(subScanSpec.getTableName());
+ log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws ExecutionSetupException {
+ this.output = output;
+ Set<MetricDTO> tables = db.getTablesFromDB();
+ if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or it's empty", subScanSpec.getTableName()));
+ }
+ this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+ try {
+ return processOpenTSDBTablesData();
+ } catch (SchemaChangeException e) {
+ log.info(e.toString());
+ return 0;
+ }
+ }
+
+ @Override
+ protected boolean isSkipQuery() {
+ return super.isSkipQuery();
+ }
+
+ @Override
+ public void close() throws Exception {
+ }
+
+ static {
+ TYPES = ImmutableMap.<OpenTSDBTypes, MinorType>builder()
+ .put(OpenTSDBTypes.STRING, MinorType.VARCHAR)
+ .put(OpenTSDBTypes.DOUBLE, MinorType.FLOAT8)
+ .put(OpenTSDBTypes.TIMESTAMP, MinorType.TIMESTAMP)
+ .build();
+ }
+
+ private static class ProjectedColumnInfo {
+ ValueVector vv;
+ ColumnDTO openTSDBColumn;
+ }
+
+ private int processOpenTSDBTablesData() throws SchemaChangeException {
+ int rowCounter = 0;
+ while (tableIterator.hasNext()) {
+ MetricDTO metricDTO = tableIterator.next();
+ rowCounter = addRowResult(metricDTO, rowCounter);
+ }
+ return rowCounter;
+ }
+
+ private int addRowResult(MetricDTO table, int rowCounter) throws SchemaChangeException {
+ setupProjectedColsIfItNull();
+ for (String time : table.getDps().keySet()) {
+ String value = table.getDps().get(time);
+ setupDataToDrillTable(table, time, value, table.getTags(), rowCounter);
+ rowCounter++;
+ }
+ return rowCounter;
+ }
+
+ private void setupProjectedColsIfItNull() throws SchemaChangeException {
+ if (projectedCols == null) {
+ initCols(new Schema(db, subScanSpec.getTableName()));
+ }
+ }
+
+ private void setupDataToDrillTable(MetricDTO table, String timestamp, String value, Map<String, String> tags, int rowCount) {
+ for (ProjectedColumnInfo pci : projectedCols) {
+ switch (pci.openTSDBColumn.getColumnName()) {
+ case "metric":
+ setStringColumnValue(table.getMetric(), pci, rowCount);
+ break;
+ case "aggregate tags":
+ setStringColumnValue(table.getAggregateTags().toString(), pci, rowCount);
+ break;
+ case "timestamp":
+ setTimestampColumnValue(timestamp, pci, rowCount);
+ break;
+ case "aggregated value":
+ setDoubleColumnValue(value, pci, rowCount);
+ break;
+ default:
+ setStringColumnValue(tags.get(pci.openTSDBColumn.getColumnName()), pci, rowCount);
+ }
+ }
+ }
+
+ private void setTimestampColumnValue(String timestamp, ProjectedColumnInfo pci, int rowCount) {
+ setTimestampColumnValue(timestamp != null ? Long.parseLong(timestamp) : Long.parseLong("0"), pci, rowCount);
--- End diff --
If timestamp column value is null, why don't we return null especially when we use nullable vectors? The same for double.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957504
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
--- End diff --
Sure, I will rename it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569087
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
+ return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set<String> getAllTableNames() {
+ return getTableNames();
+ }
+
+ @Override
+ public List<ColumnDTO> getUnfixedColumnsToSchema() {
+ Set<MetricDTO> tables = getAllMetricsByTags();
+ List<ColumnDTO> unfixedColumns = new ArrayList<>();
+
+ for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+ if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+ }
+ }
+ }
+ return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+ if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+ } else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+ }
+ }
+
+ private Set<MetricDTO> getAllMetricsByTags() {
+ try {
+ return getAllMetricsFromDBByTags();
+ } catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<String> getTableNames() {
+ try {
+ return client.getAllTablesName().execute().body();
+ } catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<MetricDTO> getAllTablesWithSpecialTag(DBQuery base) throws IOException {
+ return client.getTables(base).execute().body();
+ }
+
+ private Set<MetricDTO> getAllMetricsFromDBByTags() throws IOException {
+ Map<String, String> tags = new HashMap<>();
+
+ Query subQuery = new Query.Builder(queryParameters.get(METRIC))
+ .setAggregator(queryParameters.get(AGGREGATOR))
+ .setDownsample(queryParameters.get(DOWNSAMPLE))
+ .setTags(tags).build();
+
+ Set<Query> queries = new HashSet<>();
+ queries.add(subQuery);
+
+ DBQuery baseQuery = new DBQuery.Builder()
+ .setQueries(queries)
+ .build();
+
+ Set<MetricDTO> tables =
+ getBaseTables(baseQuery);
+
+ Set<String> extractedTags =
+ getTagsFromTables(tables);
+
+ tables.clear();
--- End diff --
Ok, I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146135322
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBGroupScan.java ---
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ListMultimap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.exec.physical.EndpointAffinity;
+import org.apache.drill.exec.physical.base.AbstractGroupScan;
+import org.apache.drill.exec.physical.base.GroupScan;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.ScanStats;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBSubScan.OpenTSDBSubScanSpec;
+import org.apache.drill.exec.store.schedule.AffinityCreator;
+import org.apache.drill.exec.store.schedule.AssignmentCreator;
+import org.apache.drill.exec.store.schedule.CompleteWork;
+import org.apache.drill.exec.store.schedule.EndpointByteMap;
+import org.apache.drill.exec.store.schedule.EndpointByteMapImpl;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@JsonTypeName("openTSDB-scan")
+public class OpenTSDBGroupScan extends AbstractGroupScan {
+
+ private static final long DEFAULT_TABLET_SIZE = 1000;
+
+ private OpenTSDBStoragePluginConfig storagePluginConfig;
+ private OpenTSDBScanSpec openTSDBScanSpec;
+ private OpenTSDBStoragePlugin storagePlugin;
+
+ private ListMultimap<Integer, OpenTSDBWork> assignments;
+ private List<SchemaPath> columns;
+ private List<OpenTSDBWork> openTSDBWorkList = Lists.newArrayList();
+ private List<EndpointAffinity> affinities;
+
+ private boolean filterPushedDown = false;
+
+ @JsonCreator
+ public OpenTSDBGroupScan(@JsonProperty("openTSDBScanSpec") OpenTSDBScanSpec openTSDBScanSpec,
+ @JsonProperty("storage") OpenTSDBStoragePluginConfig openTSDBStoragePluginConfig,
+ @JsonProperty("columns") List<SchemaPath> columns,
+ @JacksonInject StoragePluginRegistry pluginRegistry) throws IOException, ExecutionSetupException {
+ this((OpenTSDBStoragePlugin) pluginRegistry.getPlugin(openTSDBStoragePluginConfig), openTSDBScanSpec, columns);
+ }
+
+ public OpenTSDBGroupScan(OpenTSDBStoragePlugin storagePlugin,
+ OpenTSDBScanSpec scanSpec, List<SchemaPath> columns) {
+ super((String) null);
+ this.storagePlugin = storagePlugin;
+ this.storagePluginConfig = storagePlugin.getConfig();
+ this.openTSDBScanSpec = scanSpec;
+ this.columns = columns == null || columns.size() == 0 ? ALL_COLUMNS : columns;
+ init();
+ }
+
+ /**
+ * Private constructor, used for cloning.
+ *
+ * @param that The OpenTSDBGroupScan to clone
+ */
+ private OpenTSDBGroupScan(OpenTSDBGroupScan that) {
+ super((String) null);
+ this.columns = that.columns;
+ this.openTSDBScanSpec = that.openTSDBScanSpec;
+ this.storagePlugin = that.storagePlugin;
+ this.storagePluginConfig = that.storagePluginConfig;
+ this.filterPushedDown = that.filterPushedDown;
+ this.openTSDBWorkList = that.openTSDBWorkList;
+ this.assignments = that.assignments;
+ init();
+ }
+
+ private void init() {
+ Collection<DrillbitEndpoint> endpoints = storagePlugin.getContext().getBits();
+ Map<String, DrillbitEndpoint> endpointMap = Maps.newHashMap();
+
+ for (DrillbitEndpoint endpoint : endpoints) {
+ endpointMap.put(endpoint.getAddress(), endpoint);
+ }
+ }
+
+ @Override
+ public List<EndpointAffinity> getOperatorAffinity() {
+ if (affinities == null) {
+ affinities = AffinityCreator.getAffinityMap(openTSDBWorkList);
+ }
+ return affinities;
+ }
+
+ @Override
+ public int getMaxParallelizationWidth() {
+ return openTSDBWorkList.size();
+ }
+
+ @Override
+ public void applyAssignments(List<DrillbitEndpoint> incomingEndpoints) {
+ assignments = AssignmentCreator.getMappings(incomingEndpoints, openTSDBWorkList);
+ }
+
+ @Override
+ public OpenTSDBSubScan getSpecificScan(int minorFragmentId) {
+ List<OpenTSDBSubScanSpec> scanSpecList = Lists.newArrayList();
+ scanSpecList.add(new OpenTSDBSubScanSpec(getTableName()));
+ return new OpenTSDBSubScan(storagePlugin, storagePluginConfig, scanSpecList, this.columns);
+ }
+
+ @Override
+ public ScanStats getScanStats() {
+ //magic number ?
+ long recordCount = 100000L * openTSDBWorkList.size();
+ return new ScanStats(ScanStats.GroupScanProperty.NO_EXACT_ROW_COUNT, recordCount, 1, recordCount);
+ }
+
+ @Override
+ @JsonIgnore
+ public PhysicalOperator getNewWithChildren(List<PhysicalOperator> children) {
+ Preconditions.checkArgument(children.isEmpty());
+ return new OpenTSDBGroupScan(this);
+ }
+
+ @Override
+ public String getDigest() {
+ return toString();
+ }
+
+ @Override
+ @JsonIgnore
+ public boolean canPushdownProjects(List<SchemaPath> columns) {
+ return true;
+ }
+
+ @JsonIgnore
+ public String getTableName() {
+ return getOpenTSDBScanSpec().getTableName();
+ }
+
+ @JsonProperty
+ public OpenTSDBScanSpec getOpenTSDBScanSpec() {
+ return openTSDBScanSpec;
+ }
+
+ @Override
+ public GroupScan clone(List<SchemaPath> columns) {
+ OpenTSDBGroupScan newScan = new OpenTSDBGroupScan(this);
+ newScan.columns = columns;
+ return newScan;
+ }
+
+ @Override
+ public String toString() {
+ return "OpenTSDBGroupScan [OpenTSDBScanSpec="
+ + openTSDBScanSpec + "]";
+ }
+
+ private static class OpenTSDBWork implements CompleteWork {
+
+ private EndpointByteMapImpl byteMap = new EndpointByteMapImpl();
+
+ @Override
+ public long getTotalBytes() {
+ return DEFAULT_TABLET_SIZE;
--- End diff --
How `DEFAULT_TABLET_SIZE` was calculated?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147584976
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+ private final Map<String, OpenTSDBDatabaseSchema> schemaMap = Maps.newHashMap();
+
+ OpenTSDBTables(String name) {
+ super(Collections.<String>emptyList(), name);
+ }
+
+ @Override
+ public AbstractSchema getSubSchema(String name) {
+ Set<String> tables;
--- End diff --
I don' think OpenTSDB has subschemas. For example, `dfs.tmp.table`, tmp is subschema here. In OpenTSDB we refer to metric as to table but not as subschema. This method should return empty collection, he same as `getSubSchemaNames()`.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148628656
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
--- End diff --
It must be shared among several queries.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147582988
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
+public class OpenTSDBSubScan extends AbstractBase implements SubScan {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(OpenTSDBSubScan.class);
+
+ @JsonProperty
+ public final OpenTSDBStoragePluginConfig storage;
+
+ private final List<SchemaPath> columns;
+ private final OpenTSDBStoragePlugin openTSDBStoragePlugin;
+ private final List<OpenTSDBSubScanSpec> tabletScanSpecList;
+
+ @JsonCreator
+ public OpenTSDBSubScan(@JacksonInject StoragePluginRegistry registry,
+ @JsonProperty("storage") StoragePluginConfig storage,
+ @JsonProperty("tabletScanSpecList") LinkedList<OpenTSDBSubScanSpec> tabletScanSpecList,
+ @JsonProperty("columns") List<SchemaPath> columns) throws ExecutionSetupException {
+ super((String) null);
+ openTSDBStoragePlugin = (OpenTSDBStoragePlugin) registry.getPlugin(storage);
+ this.tabletScanSpecList = tabletScanSpecList;
+ this.storage = (OpenTSDBStoragePluginConfig) storage;
--- End diff --
Should consider using ` @JsonProperty("storage") OpenTSDBStoragePluginConfig storage` instead of casting.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147583417
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/query/DBQuery.java ---
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.query;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.DEFAULT_TIME;
+
+/**
+ * DBQuery is an abstraction of an openTSDB query,
+ * that used for extracting data from the storage system by POST request to DB.
+ * <p>
+ * An OpenTSDB query requires at least one sub query,
+ * a means of selecting which time series should be included in the result set.
+ */
+public class DBQuery {
+
+ /**
+ * The start time for the query. This can be a relative or absolute timestamp.
+ */
+ private String start;
+ /**
+ * One or more sub subQueries used to select the time series to return.
+ */
+ private Set<Query> queries;
+
+ private DBQuery(Builder builder) {
+ this.start = builder.start;
+ this.queries = builder.queries;
+ }
+
+ public String getStart() {
+ return start;
+ }
+
+ public Set<Query> getQueries() {
+ return queries;
+ }
+
+ public static class Builder {
+
+ private String start = DEFAULT_TIME;
+ private Set<Query> queries = new HashSet<>();
--- End diff --
So what will happen is queries object will be empty? If eventually we will fail, maybe it's better to fail in `Builder`?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r143683971
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePluginConfig.java ---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.drill.common.logical.StoragePluginConfigBase;
+
+import java.io.IOException;
+import java.util.Objects;
+
+@JsonTypeName(OpenTSDBStoragePluginConfig.NAME)
+public class OpenTSDBStoragePluginConfig extends StoragePluginConfigBase {
+
+ public static final String NAME = "openTSDB";
+
+ private final String connection;
+
+ @JsonCreator
+ public OpenTSDBStoragePluginConfig(@JsonProperty("connection") String connection) throws IOException {
+ this.connection = connection;
+ }
+
+ String getConnection() {
--- End diff --
Thanks for your remark. I will fix it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567835
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
+ List<SchemaPath> projectedColumns) throws IOException {
+ setColumns(projectedColumns);
+ this.db = client;
+ this.subScanSpec = subScanSpec;
+ db.setupQueryParameters(subScanSpec.getTableName());
+ log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws ExecutionSetupException {
+ this.output = output;
+ Set<MetricDTO> tables = db.getTablesFromDB();
+ if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or it's empty", subScanSpec.getTableName()));
+ }
+ this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+ try {
+ return processOpenTSDBTablesData();
+ } catch (SchemaChangeException e) {
+ log.info(e.toString());
--- End diff --
Sure, I will replace it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147583175
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBSubScan.java ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.exec.physical.base.AbstractBase;
+import org.apache.drill.exec.physical.base.PhysicalOperator;
+import org.apache.drill.exec.physical.base.PhysicalVisitor;
+import org.apache.drill.exec.physical.base.SubScan;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+@JsonTypeName("openTSDB-tablet-scan")
+public class OpenTSDBSubScan extends AbstractBase implements SubScan {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(OpenTSDBSubScan.class);
+
+ @JsonProperty
+ public final OpenTSDBStoragePluginConfig storage;
+
+ private final List<SchemaPath> columns;
+ private final OpenTSDBStoragePlugin openTSDBStoragePlugin;
+ private final List<OpenTSDBSubScanSpec> tabletScanSpecList;
+
+ @JsonCreator
+ public OpenTSDBSubScan(@JacksonInject StoragePluginRegistry registry,
+ @JsonProperty("storage") StoragePluginConfig storage,
+ @JsonProperty("tabletScanSpecList") LinkedList<OpenTSDBSubScanSpec> tabletScanSpecList,
+ @JsonProperty("columns") List<SchemaPath> columns) throws ExecutionSetupException {
+ super((String) null);
+ openTSDBStoragePlugin = (OpenTSDBStoragePlugin) registry.getPlugin(storage);
+ this.tabletScanSpecList = tabletScanSpecList;
+ this.storage = (OpenTSDBStoragePluginConfig) storage;
+ this.columns = columns;
+ }
+
+ public OpenTSDBSubScan(OpenTSDBStoragePlugin plugin, OpenTSDBStoragePluginConfig config,
+ List<OpenTSDBSubScanSpec> tabletInfoList, List<SchemaPath> columns) {
+ super((String) null);
+ openTSDBStoragePlugin = plugin;
+ storage = config;
+ this.tabletScanSpecList = tabletInfoList;
+ this.columns = columns;
+ }
+
+ @Override
+ public int getOperatorType() {
+ return 0;
+ }
+
+ @Override
+ public boolean isExecutable() {
+ return false;
+ }
+
+ @Override
+ public PhysicalOperator getNewWithChildren(List<PhysicalOperator> children) throws ExecutionSetupException {
+ Preconditions.checkArgument(children.isEmpty());
+ return new OpenTSDBSubScan(openTSDBStoragePlugin, storage, tabletScanSpecList, columns);
+ }
+
+ @Override
+ public Iterator<PhysicalOperator> iterator() {
+ return Collections.emptyIterator();
+ }
+
+ @Override
+ public <T, X, E extends Throwable> T accept(PhysicalVisitor<T, X, E> physicalVisitor, X value) throws E {
+ return physicalVisitor.visitSubScan(this, value);
+ }
+
+ public List<SchemaPath> getColumns() {
+ return columns;
+ }
+
+ public List<OpenTSDBSubScanSpec> getTabletScanSpecList() {
+ return tabletScanSpecList;
+ }
+
+ @JsonIgnore
+ public OpenTSDBStoragePlugin getStorageEngine() {
+ return openTSDBStoragePlugin;
+ }
+
+ @JsonIgnore
+ public OpenTSDBStoragePluginConfig getStorageConfig() {
--- End diff --
Please remove `@JsonIgnore` and rather add `@JsonProperty("storage")` and in the code above remove `@JsonProperty` above `public final OpenTSDBStoragePluginConfig storage;` for proper serde.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567854
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
+ List<SchemaPath> projectedColumns) throws IOException {
+ setColumns(projectedColumns);
+ this.db = client;
+ this.subScanSpec = subScanSpec;
+ db.setupQueryParameters(subScanSpec.getTableName());
+ log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws ExecutionSetupException {
+ this.output = output;
+ Set<MetricDTO> tables = db.getTablesFromDB();
+ if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or it's empty", subScanSpec.getTableName()));
+ }
+ this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+ try {
+ return processOpenTSDBTablesData();
+ } catch (SchemaChangeException e) {
+ log.info(e.toString());
+ return 0;
+ }
+ }
+
+ @Override
+ protected boolean isSkipQuery() {
--- End diff --
I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568941
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
--- End diff --
Sure, I will add comments for methods in the Service interface.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569383
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+ private final Map<String, OpenTSDBDatabaseSchema> schemaMap = Maps.newHashMap();
+
+ OpenTSDBTables(String name) {
+ super(Collections.<String>emptyList(), name);
+ }
+
+ @Override
+ public AbstractSchema getSubSchema(String name) {
+ Set<String> tables;
+ if (!schemaMap.containsKey(name)) {
+ tables = plugin.getClient().getAllTableNames();
+ schemaMap.put(name, new OpenTSDBDatabaseSchema(tables, this, name));
+ }
+ return schemaMap.get(name);
+ }
+
+ @Override
+ public Set<String> getSubSchemaNames() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Table getTable(String name) {
+ OpenTSDBScanSpec scanSpec = new OpenTSDBScanSpec(name);
+ name = getValidTableName(name);
+ try {
+ return new DrillOpenTSDBTable(schemaName, plugin, new Schema(plugin.getClient(), name), scanSpec);
+ } catch (Exception e) {
+ log.warn("Failure while retrieving openTSDB table {}", name, e);
+ return null;
+ }
+ }
+
+ @Override
+ public Set<String> getTableNames() {
+ return plugin.getClient().getAllTableNames();
+ }
+
+ @Override
+ public CreateTableEntry createNewTable(final String tableName, List<String> partitionColumns) {
+ return null;
--- End diff --
Ok, I will remove this overriding of the method.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148957497
--- Diff: contrib/storage-opentsdb/README.md ---
@@ -0,0 +1,64 @@
+# drill-storage-openTSDB
+
+Implementation of TSDB storage plugin. Plugin uses REST API to work with TSDB.
+
+For more information about openTSDB follow this link <http://opentsdb.net>
+
+There is list of required params:
+
+* metric - The name of a metric stored in the db.
+
+* start - The start time for the query. This can be a relative or absolute timestamp.
+
+* aggregator - The name of an aggregation function to use.
+
+optional param is:
+
+* downsample - An optional downsampling function to reduce the amount of data returned.
+
+* end - An end time for the query. If not supplied, the TSD will assume the local system time on the server.
+This may be a relative or absolute timestamp. This param is optional, and if it isn't specified we will send null
+to the db in this field, but in this case db will assume the local system time on the server.
+
+List of supported aggregators
+
+<http://opentsdb.net/docs/build/html/user_guide/query/aggregators.html>
+
+List of supported time
+
+<http://opentsdb.net/docs/build/html/user_guide/query/dates.html>
+
+Params must be specified in FROM clause of the query separated by commas. For example
+
+`openTSDB.(metric=metric_name, start=4d-ago, aggregator=sum)`
+
+Supported queries for now are listed below:
+
+```
+USE openTSDB
+```
+
+```
+SHOW tables
+```
+Will print available metrics. Max number of the printed results is a Integer.MAX value
+
+```
+SELECT * FROM openTSDB. `(metric=warp.speed.test, start=47y-ago, aggregator=sum)`
+```
+Return aggregated elements from `warp.speed.test` table since 47y-ago
+
+```
+SELECT * FROM openTSDB.`(metric=warp.speed.test, aggregator=avg, start=47y-ago)`
+```
+Return aggregated elements from `warp.speed.test` table
+
+```
+SELECT `timestamp`, sum(`aggregated value`) FROM openTSDB.`(metric=warp.speed.test, aggregator=avg, start=47y-ago)` GROUP BY `timestamp`
--- End diff --
Ok, I will add it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147584411
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
--- End diff --
final?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148934363
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/Util.java ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.base.Splitter;
+
+import java.util.Map;
+
+public class Util {
+
+ /**
+ * Parse FROM parameters to Map representation
+ *
+ * @param rowData with this syntax (metric=warp.speed.test)
+ * @return Map with params key: metric, value: warp.speed.test
+ */
+ public static Map<String, String> parseFromRowData(String rowData) {
+ String FROMRowData = rowData.replaceAll("[()]", "");
--- End diff --
`fromRowData `
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568915
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
--- End diff --
Sure, it must be final.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569604
--- Diff: distribution/pom.xml ---
@@ -190,11 +190,21 @@
</dependency>
<dependency>
<groupId>org.apache.drill.contrib</groupId>
+ <artifactId>drill-opentsdb-storage</artifactId>
--- End diff --
This is merge mistake. I will solve this problem.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147585085
--- Diff: contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestDataHolder.java ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+class TestDataHolder {
--- End diff --
public
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568031
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+ private final DrillbitContext context;
+
+ private final OpenTSDBStoragePluginConfig engineConfig;
+ private final OpenTSDBSchemaFactory schemaFactory;
+
+ private final ServiceImpl db;
+
+ public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration, DrillbitContext context, String name) throws IOException {
+ this.context = context;
+ this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+ this.engineConfig = configuration;
+ this.db = new ServiceImpl("http://" + configuration.getConnection());
+ }
+
+ @Override
+ public void start() throws IOException {
+ }
+
+ @Override
+ public void close() throws Exception {
+ }
+
+ @Override
+ public boolean supportsRead() {
+ return true;
+ }
+
+ @Override
+ public OpenTSDBStoragePluginConfig getConfig() {
+ return engineConfig;
+ }
+
+ @Override
+ public OpenTSDBGroupScan getPhysicalScan(String userName, JSONOptions selection) throws IOException {
+ OpenTSDBScanSpec scanSpec = selection.getListWith(new ObjectMapper(), new TypeReference<OpenTSDBScanSpec>() {
+ });
+ return new OpenTSDBGroupScan(this, scanSpec, null);
+ }
+
+ @Override
+ public boolean supportsWrite() {
+ return true;
--- End diff --
Ok, I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569508
--- Diff: contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestDataHolder.java ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+class TestDataHolder {
+
+ static final String SAMPLE_DATA_FOR_POST_REQUEST_WITH_TAGS = "[{" +
--- End diff --
Sure.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148934421
--- Diff: contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestOpenTSDBPlugin.java ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.apache.drill.PlanTestBase;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.apache.drill.exec.store.StoragePluginRegistry;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson;
+import static com.github.tomakehurst.wiremock.client.WireMock.get;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;
+import static org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WTIHOUT_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITHOUT_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITH_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.REQUEST_TO_NONEXISTENT_METRIC;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_NAME_REQUEST;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_REQUEST;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITHOUT_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_REQUEST_WITH_TAGS;
+
+public class TestOpenTSDBPlugin extends PlanTestBase {
+
+ protected static OpenTSDBStoragePlugin storagePlugin;
+ protected static OpenTSDBStoragePluginConfig storagePluginConfig;
+
+ @Rule
+ public WireMockRule wireMockRule = new WireMockRule(10000);
+
+ @BeforeClass
+ public static void addTestDataToDB() throws Exception {
--- End diff --
Please rename -> `setup`
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569566
--- Diff: contrib/storage-opentsdb/src/test/resources/bootstrap-storage-plugins.json ---
@@ -0,0 +1,9 @@
+{
+ "storage": {
+ openTSDB: {
+ type: "openTSDB",
+ connection: "localhost:8089",
--- End diff --
Sure.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567934
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBScanSpec.java ---
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public class OpenTSDBScanSpec {
+
+ private final String tableName;
+
+ @JsonCreator
+ public OpenTSDBScanSpec(@JsonProperty("tableName") String tableName) {
+ this.tableName = tableName;
+ }
+
+ public String getTableName() {
+ return tableName;
+ }
--- End diff --
Sure, I will add `toString()`
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147582378
--- Diff: contrib/storage-opentsdb/README.md ---
@@ -0,0 +1 @@
+# drill-storage-openTSDB
--- End diff --
It would be nice if you add some information how to query data, etc, similar that is added in the pull request description.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147582692
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
+ List<SchemaPath> projectedColumns) throws IOException {
+ setColumns(projectedColumns);
+ this.db = client;
+ this.subScanSpec = subScanSpec;
+ db.setupQueryParameters(subScanSpec.getTableName());
+ log.debug("Scan spec: {}", subScanSpec);
+ }
+
+ @Override
+ public void setup(OperatorContext context, OutputMutator output) throws ExecutionSetupException {
+ this.output = output;
+ Set<MetricDTO> tables = db.getTablesFromDB();
+ if (tables == null || tables.isEmpty()) {
+ throw new ValidationError(String.format("Table '%s' not found or it's empty", subScanSpec.getTableName()));
+ }
+ this.tableIterator = tables.iterator();
+ }
+
+ @Override
+ public int next() {
+ try {
+ return processOpenTSDBTablesData();
+ } catch (SchemaChangeException e) {
+ log.info(e.toString());
+ return 0;
+ }
+ }
+
+ @Override
+ protected boolean isSkipQuery() {
--- End diff --
Do you need to override this method as you just call super inside?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567538
--- Diff: contrib/storage-opentsdb/README.md ---
@@ -0,0 +1 @@
+# drill-storage-openTSDB
--- End diff --
Sure, I will update README with more detailed info.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146134226
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
+ return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set<String> getAllTableNames() {
+ return getTableNames();
+ }
+
+ @Override
+ public List<ColumnDTO> getUnfixedColumnsToSchema() {
+ Set<MetricDTO> tables = getAllMetricsByTags();
+ List<ColumnDTO> unfixedColumns = new ArrayList<>();
+
+ for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+ if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+ }
+ }
+ }
+ return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+ if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+ } else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+ }
+ }
+
+ private Set<MetricDTO> getAllMetricsByTags() {
+ try {
+ return getAllMetricsFromDBByTags();
+ } catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<String> getTableNames() {
+ try {
+ return client.getAllTablesName().execute().body();
+ } catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
--- End diff --
1. Why do we return empty collection on error?
2. `e.printStackTrace();`?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r147582647
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
--- End diff --
public?
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569320
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/schema/OpenTSDBSchemaFactory.java ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.schema;
+
+import com.google.common.collect.Maps;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.schema.Table;
+import org.apache.drill.exec.planner.logical.CreateTableEntry;
+import org.apache.drill.exec.planner.logical.DrillTable;
+import org.apache.drill.exec.planner.logical.DynamicDrillTable;
+import org.apache.drill.exec.store.AbstractSchema;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.SchemaFactory;
+import org.apache.drill.exec.store.openTSDB.DrillOpenTSDBTable;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePlugin;
+import org.apache.drill.exec.store.openTSDB.OpenTSDBStoragePluginConfig;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Util.getValidTableName;
+
+public class OpenTSDBSchemaFactory implements SchemaFactory {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBSchemaFactory.class);
+
+ private final String schemaName;
+ private OpenTSDBStoragePlugin plugin;
+
+ public OpenTSDBSchemaFactory(OpenTSDBStoragePlugin plugin, String schemaName) {
+ this.plugin = plugin;
+ this.schemaName = schemaName;
+ }
+
+ @Override
+ public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent) throws IOException {
+ OpenTSDBTables schema = new OpenTSDBTables(schemaName);
+ parent.add(schemaName, schema);
+ }
+
+ class OpenTSDBTables extends AbstractSchema {
+ private final Map<String, OpenTSDBDatabaseSchema> schemaMap = Maps.newHashMap();
+
+ OpenTSDBTables(String name) {
+ super(Collections.<String>emptyList(), name);
+ }
+
+ @Override
+ public AbstractSchema getSubSchema(String name) {
+ Set<String> tables;
--- End diff --
Ok, I will change it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148567743
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBRecordReader.java ---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.logical.ValidationError;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.exception.SchemaChangeException;
+import org.apache.drill.exec.expr.TypeHelper;
+import org.apache.drill.exec.ops.OperatorContext;
+import org.apache.drill.exec.physical.impl.OutputMutator;
+import org.apache.drill.exec.record.MaterializedField;
+import org.apache.drill.exec.store.AbstractRecordReader;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Schema;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.apache.drill.exec.vector.NullableFloat8Vector;
+import org.apache.drill.exec.vector.NullableTimeStampVector;
+import org.apache.drill.exec.vector.NullableVarCharVector;
+import org.apache.drill.exec.vector.ValueVector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+public class OpenTSDBRecordReader extends AbstractRecordReader {
+
+ private static final Logger log = LoggerFactory.getLogger(OpenTSDBRecordReader.class);
+
+ private static final Map<OpenTSDBTypes, MinorType> TYPES;
+
+ private Service db;
+
+ private Iterator<MetricDTO> tableIterator;
+ private OutputMutator output;
+ private ImmutableList<ProjectedColumnInfo> projectedCols;
+ private OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec;
+
+ OpenTSDBRecordReader(Service client, OpenTSDBSubScan.OpenTSDBSubScanSpec subScanSpec,
--- End diff --
Ok, I will add access modifier.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148568016
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBStoragePlugin.java ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.drill.common.JSONOptions;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.store.AbstractStoragePlugin;
+import org.apache.drill.exec.store.SchemaConfig;
+import org.apache.drill.exec.store.openTSDB.client.services.ServiceImpl;
+import org.apache.drill.exec.store.openTSDB.schema.OpenTSDBSchemaFactory;
+
+import java.io.IOException;
+
+public class OpenTSDBStoragePlugin extends AbstractStoragePlugin {
+
+ private final DrillbitContext context;
+
+ private final OpenTSDBStoragePluginConfig engineConfig;
+ private final OpenTSDBSchemaFactory schemaFactory;
+
+ private final ServiceImpl db;
+
+ public OpenTSDBStoragePlugin(OpenTSDBStoragePluginConfig configuration, DrillbitContext context, String name) throws IOException {
+ this.context = context;
+ this.schemaFactory = new OpenTSDBSchemaFactory(this, name);
+ this.engineConfig = configuration;
+ this.db = new ServiceImpl("http://" + configuration.getConnection());
+ }
+
+ @Override
+ public void start() throws IOException {
--- End diff --
Sure, I will remove it.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146133090
--- Diff: exec/java-exec/src/test/resources/drill-module.conf ---
@@ -10,7 +10,7 @@ drill: {
packages += "org.apache.drill.exec.testing",
packages += "org.apache.drill.exec.rpc.user.security.testing"
}
- test.query.printing.silent : false,
+ test.query.printing.silent : false,
--- End diff --
I guess change in this file is not required...
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r146135774
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/OpenTSDBScanSpec.java ---
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public class OpenTSDBScanSpec {
+
+ private final String tableName;
+
+ @JsonCreator
+ public OpenTSDBScanSpec(@JsonProperty("tableName") String tableName) {
+ this.tableName = tableName;
+ }
+
+ public String getTableName() {
+ return tableName;
+ }
--- End diff --
If you'll do expain plan for the query:
```
0: jdbc:drill:drillbit=localhost> explain plan for SELECT * FROM openTSDB.`(metric=mymetric.stock)`;
+------+------+
| text | json |
+------+------+
| 00-00 Screen
00-01 Project(metric=[$0], aggregate tags=[$1], timestamp=[$2], aggregated value=[$3], symbol=[$4])
00-02 Scan(groupscan=[OpenTSDBGroupScan [OpenTSDBScanSpec=org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec@57979642]])
| {
"head" : {
"version" : 1,
"generator" : {
"type" : "ExplainHandler",
"info" : ""
},
"type" : "APACHE_DRILL_PHYSICAL",
"options" : [ ],
"queue" : 0,
"hasResourcePlan" : false,
"resultMode" : "EXEC"
},
"graph" : [ {
"pop" : "openTSDB-scan",
"@id" : 2,
"openTSDBScanSpec" : {
"tableName" : "(metric=mymetric.stock)"
},
"cost" : 0.0
}, {
"pop" : "project",
"@id" : 1,
"exprs" : [ {
"ref" : "`metric`",
"expr" : "`metric`"
}, {
"ref" : "`aggregate tags`",
"expr" : "`aggregate tags`"
}, {
"ref" : "`timestamp`",
"expr" : "`timestamp`"
}, {
"ref" : "`aggregated value`",
"expr" : "`aggregated value`"
}, {
"ref" : "`symbol`",
"expr" : "`symbol`"
} ],
"child" : 2,
"outputProj" : true,
"initialAllocation" : 1000000,
"maxAllocation" : 10000000000,
"cost" : 1.0
}, {
"pop" : "screen",
"@id" : 0,
"child" : 1,
"initialAllocation" : 1000000,
"maxAllocation" : 10000000000,
"cost" : 1.0
} ]
} |
+------+------+
```
you'll see `OpenTSDBScanSpec=org.apache.drill.exec.store.openTSDB.OpenTSDBScanSpec@57979642`.
I guess we might want to add `toString()` method in this class.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569173
--- Diff: contrib/storage-opentsdb/src/main/java/org/apache/drill/exec/store/openTSDB/client/services/ServiceImpl.java ---
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.openTSDB.client.services;
+
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDB;
+import org.apache.drill.exec.store.openTSDB.client.OpenTSDBTypes;
+import org.apache.drill.exec.store.openTSDB.client.Service;
+import org.apache.drill.exec.store.openTSDB.client.query.DBQuery;
+import org.apache.drill.exec.store.openTSDB.client.query.Query;
+import org.apache.drill.exec.store.openTSDB.dto.ColumnDTO;
+import org.apache.drill.exec.store.openTSDB.dto.MetricDTO;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import retrofit2.Retrofit;
+import retrofit2.converter.jackson.JacksonConverterFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.drill.exec.store.openTSDB.Constants.AGGREGATOR;
+import static org.apache.drill.exec.store.openTSDB.Constants.DOWNSAMPLE;
+import static org.apache.drill.exec.store.openTSDB.Constants.METRIC;
+import static org.apache.drill.exec.store.openTSDB.Util.isTableNameValid;
+import static org.apache.drill.exec.store.openTSDB.Util.parseFROMRowData;
+
+public class ServiceImpl implements Service {
+
+ private static final Logger log =
+ LoggerFactory.getLogger(ServiceImpl.class);
+
+ private OpenTSDB client;
+ private Map<String, String> queryParameters;
+
+ public ServiceImpl(String connectionURL) {
+ this.client = new Retrofit.Builder()
+ .baseUrl(connectionURL)
+ .addConverterFactory(JacksonConverterFactory.create())
+ .build()
+ .create(OpenTSDB.class);
+ }
+
+ @Override
+ public Set<MetricDTO> getTablesFromDB() {
+ return getAllMetricsByTags();
+ }
+
+ @Override
+ public Set<String> getAllTableNames() {
+ return getTableNames();
+ }
+
+ @Override
+ public List<ColumnDTO> getUnfixedColumnsToSchema() {
+ Set<MetricDTO> tables = getAllMetricsByTags();
+ List<ColumnDTO> unfixedColumns = new ArrayList<>();
+
+ for (MetricDTO table : tables) {
+ for (String tag : table.getTags().keySet()) {
+ ColumnDTO tmp = new ColumnDTO(tag, OpenTSDBTypes.STRING);
+ if (!unfixedColumns.contains(tmp)) {
+ unfixedColumns.add(tmp);
+ }
+ }
+ }
+ return unfixedColumns;
+ }
+
+ @Override
+ public void setupQueryParameters(String rowData) {
+ if (!isTableNameValid(rowData)) {
+ this.queryParameters = parseFROMRowData(rowData);
+ } else {
+ Map<String, String> params = new HashMap<>();
+ params.put(METRIC, rowData);
+ this.queryParameters = params;
+ }
+ }
+
+ private Set<MetricDTO> getAllMetricsByTags() {
+ try {
+ return getAllMetricsFromDBByTags();
+ } catch (IOException e) {
+ logIOException(e);
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<String> getTableNames() {
+ try {
+ return client.getAllTablesName().execute().body();
+ } catch (IOException e) {
+ e.printStackTrace();
+ return Collections.emptySet();
+ }
+ }
+
+ private Set<MetricDTO> getAllTablesWithSpecialTag(DBQuery base) throws IOException {
+ return client.getTables(base).execute().body();
+ }
+
+ private Set<MetricDTO> getAllMetricsFromDBByTags() throws IOException {
+ Map<String, String> tags = new HashMap<>();
+
+ Query subQuery = new Query.Builder(queryParameters.get(METRIC))
+ .setAggregator(queryParameters.get(AGGREGATOR))
+ .setDownsample(queryParameters.get(DOWNSAMPLE))
+ .setTags(tags).build();
+
+ Set<Query> queries = new HashSet<>();
+ queries.add(subQuery);
+
+ DBQuery baseQuery = new DBQuery.Builder()
+ .setQueries(queries)
+ .build();
+
+ Set<MetricDTO> tables =
+ getBaseTables(baseQuery);
+
+ Set<String> extractedTags =
+ getTagsFromTables(tables);
+
+ tables.clear();
+
+ for (String value : extractedTags) {
+ transformTagsForRequest(tags, value);
+ tables.addAll(getAllTablesWithSpecialTag(baseQuery));
+ }
+ return tables;
+ }
+
+ private Set<MetricDTO> getBaseTables(DBQuery base) throws IOException {
+ return getAllTablesWithSpecialTag(base);
+ }
+
+ private Set<String> getTagsFromTables(Set<MetricDTO> tables) {
+ Set<String> extractedTags = new HashSet<>();
+
+ for (MetricDTO table : tables) {
+ extractedTags.addAll(table.getAggregateTags());
+ extractedTags.addAll(table.getTags().keySet());
+ }
+
+ return extractedTags;
+ }
+
+ private void transformTagsForRequest(Map<String, String> tags, String value) {
+ tags.clear();
+ tags.put(value, "*");
+ }
+
+ private void logIOException(IOException e) {
+ log.warn("A problem occurred when talking to the server", e);
--- End diff --
Sure, I will throw an exception with a clear message.
---
[GitHub] drill issue #774: DRILL-5337: OpenTSDB storage plugin
Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/774
+1, LGTM.
---
[GitHub] drill pull request #774: DRILL-5337: OpenTSDB storage plugin
Posted by Vlad-Storona <gi...@git.apache.org>.
Github user Vlad-Storona commented on a diff in the pull request:
https://github.com/apache/drill/pull/774#discussion_r148569546
--- Diff: contrib/storage-opentsdb/src/test/java/org/apache/drill/store/openTSDB/TestOpenTSDBPlugin.java ---
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.store.openTSDB;
+
+import com.github.tomakehurst.wiremock.junit.WireMockRule;
+import org.apache.drill.BaseTestQuery;
+import org.apache.drill.common.exceptions.UserRemoteException;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
+import static com.github.tomakehurst.wiremock.client.WireMock.equalToJson;
+import static com.github.tomakehurst.wiremock.client.WireMock.get;
+import static com.github.tomakehurst.wiremock.client.WireMock.post;
+import static com.github.tomakehurst.wiremock.client.WireMock.stubFor;
+import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;
+import static org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.DOWNSAMPLE_REQUEST_WTIHOUT_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITHOUT_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.POST_REQUEST_WITH_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.REQUEST_TO_NONEXISTENT_METRIC;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_NAME_REQUEST;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_GET_TABLE_REQUEST;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITHOUT_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_DOWNSAMPLE_REQUEST_WITH_TAGS;
+import static org.apache.drill.store.openTSDB.TestDataHolder.SAMPLE_DATA_FOR_POST_REQUEST_WITH_TAGS;
+
+public class TestOpenTSDBPlugin extends BaseTestQuery {
--- End diff --
This type of query unsupported in this realization.
---