You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2020/02/12 07:54:32 UTC
[GitHub] [ignite] pavlukhin opened a new pull request #7412: IGNITE-12067
User SQL query execution metrics
pavlukhin opened a new pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] pavlukhin merged pull request #7412: IGNITE-12067 User SQL
query execution metrics
Posted by GitBox <gi...@apache.org>.
pavlukhin merged pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] AMashenkov commented on a change in pull request #7412:
IGNITE-12067 User SQL query execution metrics
Posted by GitBox <gi...@apache.org>.
AMashenkov commented on a change in pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412#discussion_r378120632
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/query/RunningQueryManager.java
##########
@@ -72,6 +77,18 @@
/** Query history tracker. */
private volatile QueryHistoryTracker qryHistTracker;
+ /** Number of successfully executed queries. */
+ private final LongAdderMetric successQrsCnt;
Review comment:
Why LongAdderMetric is used for succeeded queries and AtomicLongMetric is used for failed\canceled ones?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] AMashenkov commented on a change in pull request #7412:
IGNITE-12067 User SQL query execution metrics
Posted by GitBox <gi...@apache.org>.
AMashenkov commented on a change in pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412#discussion_r378114397
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/bulkload/BulkLoadProcessor.java
##########
@@ -100,27 +103,36 @@ public void processBatch(byte[] batchData, boolean isLastBatch) throws IgniteChe
}
}
+ /**
+ * Is called to notify processor, that bulk load execution, this processor is performing, failed with specified
Review comment:
* Failure callback.
* @param failReason Exception caused a failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] AMashenkov commented on a change in pull request #7412:
IGNITE-12067 User SQL query execution metrics
Posted by GitBox <gi...@apache.org>.
AMashenkov commented on a change in pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412#discussion_r378118330
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/odbc/jdbc/JdbcBulkLoadProcessor.java
##########
@@ -142,6 +142,15 @@ public void processBatch(JdbcBulkLoadBatchRequest req)
}
}
+ /**
+ * Gets notified if current bulk load failed.
Review comment:
Failure callback.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] AMashenkov commented on a change in pull request #7412:
IGNITE-12067 User SQL query execution metrics
Posted by GitBox <gi...@apache.org>.
AMashenkov commented on a change in pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412#discussion_r378113116
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/bulkload/BulkLoadProcessor.java
##########
@@ -51,6 +51,9 @@
/** Query id. */
private final Long qryId;
+ /** Exception, current load process ended with, or {@code null} if in progress or if succeded. */
+ private Exception failReason;
Review comment:
Load failure reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] AMashenkov commented on a change in pull request #7412:
IGNITE-12067 User SQL query execution metrics
Posted by GitBox <gi...@apache.org>.
AMashenkov commented on a change in pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412#discussion_r378118330
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/odbc/jdbc/JdbcBulkLoadProcessor.java
##########
@@ -142,6 +142,15 @@ public void processBatch(JdbcBulkLoadBatchRequest req)
}
}
+ /**
+ * Gets notified if current bulk load failed.
Review comment:
Failure callback.
@param failReason Exception caused a failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] AMashenkov commented on a change in pull request #7412:
IGNITE-12067 User SQL query execution metrics
Posted by GitBox <gi...@apache.org>.
AMashenkov commented on a change in pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412#discussion_r378123620
##########
File path: modules/indexing/src/test/java/org/apache/ignite/internal/metric/SqlStatisticsAbstractTest.java
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metric;
+
+import java.util.Collections;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.cache.QueryEntity;
+import org.apache.ignite.cache.query.annotations.QuerySqlFunction;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.testframework.GridTestUtils;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+
+/**
+ * Abstract test for sql metrics tests.
+ */
+public class SqlStatisticsAbstractTest extends GridCommonAbstractTest {
+ /**
+ * Timeout for each wait on sync operation in seconds.
+ */
+ public static final long WAIT_OP_TIMEOUT_SEC = 15;
+
+ /**
+ * Number of rows in the test table.
+ */
+ public static final int TABLE_SIZE = 10_000;
+
+ /**
+ * Start the cache with a test table and test data.
+ */
+ protected IgniteCache createCacheFrom(Ignite node) {
+ CacheConfiguration<Integer, String> ccfg = new CacheConfiguration<Integer, String>(DEFAULT_CACHE_NAME)
+ .setSqlFunctionClasses(SuspendQuerySqlFunctions.class)
+ .setQueryEntities(Collections.singleton(
+ new QueryEntity(Integer.class.getName(), String.class.getName())
+ .setTableName("TAB")
+ .addQueryField("id", Integer.class.getName(), null)
+ .addQueryField("name", String.class.getName(), null)
+ .setKeyFieldName("id")
+ .setValueFieldName("name")
+ ));
+
+ IgniteCache<Integer, String> cache = node.createCache(ccfg);
+
+ try (IgniteDataStreamer<Object, Object> ds = node.dataStreamer(DEFAULT_CACHE_NAME)) {
+ for (int i = 0; i < TABLE_SIZE; i++)
+ ds.addData(i, UUID.randomUUID().toString());
+ }
+
+ return cache;
+ }
+
+ /**
+ * Run async action and log if exception occured.
+ *
+ * @param act action to perform on other thread.
+ * @return future object to "action complited" event.
+ */
+ protected IgniteInternalFuture runAsyncX(Runnable act) {
+ return GridTestUtils.runAsync(() -> {
+ try {
+ act.run();
+ }
+ catch (Throwable th) {
+ log.error("Failed to perform async action. Probably test is broken.", th);
+ }
+ });
+ }
+
+ /**
+ * This class exports function to the sql engine. Function implementation allows us to suspend query execution on test
+ * logic condition.
+ */
+ public static class SuspendQuerySqlFunctions {
+ /**
+ * How many rows should be processed (by all nodes in total)
+ */
+ private static final int DFLT_PROCESS_ROWS_TO_SUSPEND = TABLE_SIZE / 4;
+
+ /**
+ * Latch to await till full scan query that uses this class function have done some job, so some memory is
Review comment:
Next statement makes no sense here "so some memory is reserved."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] AMashenkov commented on a change in pull request #7412:
IGNITE-12067 User SQL query execution metrics
Posted by GitBox <gi...@apache.org>.
AMashenkov commented on a change in pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412#discussion_r378121815
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/query/RunningQueryManager.java
##########
@@ -72,6 +77,18 @@
/** Query history tracker. */
private volatile QueryHistoryTracker qryHistTracker;
+ /** Number of successfully executed queries. */
+ private final LongAdderMetric successQrsCnt;
+
+ /** Number of failed queries in total by any reason. */
+ private final AtomicLongMetric failedQrsCnt;
+
+ /**
+ * Number of canceled queries. Canceled queries a treated as failed and counting twice: here and in {@link
Review comment:
Cancelled queries count metric.
Note: Cancelled query is treated as failed one as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] AMashenkov commented on a change in pull request #7412:
IGNITE-12067 User SQL query execution metrics
Posted by GitBox <gi...@apache.org>.
AMashenkov commented on a change in pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412#discussion_r378120632
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/query/RunningQueryManager.java
##########
@@ -72,6 +77,18 @@
/** Query history tracker. */
private volatile QueryHistoryTracker qryHistTracker;
+ /** Number of successfully executed queries. */
+ private final LongAdderMetric successQrsCnt;
Review comment:
I'm just curious.
Why LongAdderMetric is used for succeeded queries and AtomicLongMetric is used for failed\canceled ones?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [ignite] AMashenkov commented on a change in pull request #7412:
IGNITE-12067 User SQL query execution metrics
Posted by GitBox <gi...@apache.org>.
AMashenkov commented on a change in pull request #7412: IGNITE-12067 User SQL query execution metrics
URL: https://github.com/apache/ignite/pull/7412#discussion_r378122349
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/query/RunningQueryManager.java
##########
@@ -72,6 +77,18 @@
/** Query history tracker. */
private volatile QueryHistoryTracker qryHistTracker;
+ /** Number of successfully executed queries. */
+ private final LongAdderMetric successQrsCnt;
+
+ /** Number of failed queries in total by any reason. */
Review comment:
Failed and cancelled queries count metric.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services