You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/05/04 22:15:00 UTC

[jira] [Commented] (QPID-8581) [Broker-J] Broker-J Query REST API improvements

    [ https://issues.apache.org/jira/browse/QPID-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531969#comment-17531969 ] 

ASF GitHub Bot commented on QPID-8581:
--------------------------------------

alex-rufous commented on code in PR #121:
URL: https://github.com/apache/qpid-broker-j/pull/121#discussion_r865411418


##########
broker-core/src/main/java/org/apache/qpid/server/model/port/HttpPort.java:
##########
@@ -72,6 +72,21 @@
     @ManagedContextDefault(name = TLS_SESSION_CACHE_SIZE, description = "TLS session cache size for HTTP ports.")
     int DEFAULT_TLS_SESSION_CACHE_SIZE = 1000;
 
+    String QUERY_ENGINE_CACHE_SIZE = "qpid.port.http.query.engine.cacheSize";
+    @SuppressWarnings("unused")
+    @ManagedContextDefault(name = QUERY_ENGINE_CACHE_SIZE, description = "Broker query engine cache size.")
+    int DEFAULT_QUERY_ENGINE_CACHE_SIZE = 1000;
+
+    String QUERY_ENGINE_MAX_QUERY_DEPTH = "qpid.port.http.query.engine.maxQueryDepth";
+    @SuppressWarnings("unused")
+    @ManagedContextDefault(name = QUERY_ENGINE_MAX_QUERY_DEPTH, description = "Broker query engine max query depth.")
+    int DEFAULT_QUERY_ENGINE_MAX_QUERY_DEPTH = 4096;
+
+    String QUERY_ENGINE_ZONE_ID = "qpid.port.http.query.engine.zoneId";
+    @SuppressWarnings("unused")
+    @ManagedContextDefault(name = QUERY_ENGINE_ZONE_ID, description = "Broker query engine zone id.")

Review Comment:
   A word 'time' is missed before word 'zone'.  IMHO, it should be 'Broker query engine time zone id.'



##########
broker-plugins/broker-query-engine/src/main/java/org/apache/qpid/server/query/engine/evaluator/EvaluationContext.java:
##########
@@ -0,0 +1,154 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.qpid.server.query.engine.evaluator;
+
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.qpid.server.query.engine.parsing.expression.Expression;
+import org.apache.qpid.server.query.engine.parsing.query.QueryExpression;
+
+/**
+ * Holds temporary values needed during query evaluation
+ */
+@SuppressWarnings({"java:S116", "unchecked"})
+public class EvaluationContext
+{
+    public static final String BROKER = "broker";
+
+    public static final String STATISTICS = "statistics";
+
+    public static final String QUERY_AGGREGATED_RESULT = "query.aggregated.result";
+
+    public static final String QUERY_ALIASES = "query.aliases";
+
+    public static final String QUERY_DATETIME_PATTERN_OVERRIDEN = "query.datetime.pattern.overriden";
+
+    public static final String QUERY_DEPTH = "query.depth";
+
+    public static final String QUERY_ORDERING = "query.ordering";
+
+    public static final String QUERY_ITEMS_FOR_REMOVAL = "query.items.for.removal";
+
+    public static final String QUERY_ORDER_ITEMS_FOR_REMOVAL = "query.order.items.for.removal";
+
+    public static final String QUERY_SETTINGS = "query.settings";
+
+    public static final String COMPARATORS = "comparators";
+
+    private final Map<Object, Object> _values = new HashMap<>();

Review Comment:
   HashMap is not thread safe. Would it be better to use ConcurrentHashMap? I am not sure whether thread safety is need for this class



##########
broker-plugins/broker-query-engine/src/main/java/org/apache/qpid/server/query/engine/QueryEngine.java:
##########
@@ -0,0 +1,129 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.qpid.server.query.engine;
+
+import java.math.BigDecimal;
+import java.time.ZoneId;
+import java.util.Map;
+import java.util.Objects;
+
+import org.apache.qpid.server.model.Broker;
+import org.apache.qpid.server.model.port.HttpPort;
+import org.apache.qpid.server.query.engine.cache.MaxSizeHashMap;
+import org.apache.qpid.server.query.engine.evaluator.QueryEvaluator;
+import org.apache.qpid.server.query.engine.evaluator.settings.QuerySettings;
+import org.apache.qpid.server.query.engine.parsing.query.QueryExpression;
+
+/**
+ * Bridge class between broker configuration and query evaluator
+ */
+// sonar complains about underscores in variable names
+@SuppressWarnings("java:S116")
+public class QueryEngine
+{
+    /**
+     * Broker instance
+     */
+    private final Broker<?> _broker;
+
+    /**
+     * Cache holding queries
+     */
+    private final Map<String, QueryExpression<?, ?>> _queryCache;
+
+    /**
+     * Maximal allowed BigDecimal value.
+     * Is needed to prevent heap memory consumption when calculating very large numbers.
+     */
+    private BigDecimal _maxBigDecimalValue = BigDecimal.valueOf(Double.MAX_VALUE).pow(4);
+
+    /**
+     * Maximal amount of queries allowed caching
+     */
+    private final int _maxQueryCacheSize;
+
+    /**
+     * Maximal amount of query tree nodes allowed
+     */
+    private int _maxQueryDepth;
+
+    /**
+     * Zone id
+     */
+    private ZoneId _zoneId;
+
+    /**
+     * Constructor injects broker and retrieves default configuration values
+     *
+     * @param broker Broker instance
+     */
+    // mutable broker instance is stored intentionally
+    @SuppressWarnings("findbugs:EI_EXPOSE_REP2")
+    public QueryEngine(final Broker<?> broker)
+    {
+        Objects.requireNonNull(broker, "Broker instance not provided for querying");
+        _broker = broker;
+        final HttpPort<?> httpPort = broker.getPorts().stream()

Review Comment:
   In case when multiple http ports with different query context variables are declared, with the current implementation the caching settings can be taken from the first port. IMHO, either the cache instance should be  created per port or  you can move the cache settings into http management or broker. Are you planning to create a cache instance per port?



##########
broker-plugins/broker-query-engine/src/main/java/org/apache/qpid/server/query/engine/QueryEngine.java:
##########
@@ -0,0 +1,129 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.qpid.server.query.engine;
+
+import java.math.BigDecimal;
+import java.time.ZoneId;
+import java.util.Map;
+import java.util.Objects;
+
+import org.apache.qpid.server.model.Broker;
+import org.apache.qpid.server.model.port.HttpPort;
+import org.apache.qpid.server.query.engine.cache.MaxSizeHashMap;
+import org.apache.qpid.server.query.engine.evaluator.QueryEvaluator;
+import org.apache.qpid.server.query.engine.evaluator.settings.QuerySettings;
+import org.apache.qpid.server.query.engine.parsing.query.QueryExpression;
+
+/**
+ * Bridge class between broker configuration and query evaluator
+ */
+// sonar complains about underscores in variable names
+@SuppressWarnings("java:S116")
+public class QueryEngine
+{
+    /**
+     * Broker instance
+     */
+    private final Broker<?> _broker;
+
+    /**
+     * Cache holding queries
+     */
+    private final Map<String, QueryExpression<?, ?>> _queryCache;
+
+    /**
+     * Maximal allowed BigDecimal value.
+     * Is needed to prevent heap memory consumption when calculating very large numbers.
+     */
+    private BigDecimal _maxBigDecimalValue = BigDecimal.valueOf(Double.MAX_VALUE).pow(4);
+
+    /**
+     * Maximal amount of queries allowed caching
+     */
+    private final int _maxQueryCacheSize;
+
+    /**
+     * Maximal amount of query tree nodes allowed
+     */
+    private int _maxQueryDepth;
+
+    /**
+     * Zone id
+     */
+    private ZoneId _zoneId;
+
+    /**
+     * Constructor injects broker and retrieves default configuration values
+     *
+     * @param broker Broker instance
+     */
+    // mutable broker instance is stored intentionally
+    @SuppressWarnings("findbugs:EI_EXPOSE_REP2")
+    public QueryEngine(final Broker<?> broker)
+    {
+        Objects.requireNonNull(broker, "Broker instance not provided for querying");
+        _broker = broker;
+        final HttpPort<?> httpPort = broker.getPorts().stream()
+            .filter(port -> Objects.equals("HTTP", port.getType()))
+            .map(port -> (HttpPort<?>)port)
+            .findFirst()
+            .orElseThrow(() -> new IllegalArgumentException("HTTP port not found"));

Review Comment:
   Theoretically, the Broker can be ran without HTTP port. If exception is thrown, it might crash the Broker. IMHO, the cache should not be created in such case.



##########
broker-plugins/broker-query-engine/src/main/java/org/apache/qpid/server/query/engine/parsing/converter/NumberConverter.java:
##########
@@ -0,0 +1,250 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.qpid.server.query.engine.parsing.converter;
+
+import java.math.BigDecimal;
+import java.math.RoundingMode;
+import java.util.Map;
+import java.util.function.Function;
+
+import com.google.common.collect.ImmutableMap;
+
+import org.apache.qpid.server.query.engine.evaluator.EvaluationContext;
+import org.apache.qpid.server.query.engine.evaluator.EvaluationContextHolder;
+import org.apache.qpid.server.query.engine.evaluator.settings.QuerySettings;
+import org.apache.qpid.server.query.engine.exception.Errors;
+import org.apache.qpid.server.query.engine.exception.QueryParsingException;
+import org.apache.qpid.server.query.engine.parsing.utils.StringUtils;
+
+/**
+ * Utility class for numeric conversions
+ */
+public final class NumberConverter
+{
+    /**
+     * Conversion rules
+     */
+    private static final Map<Class<?>, Map<Class<?>, Function<Object, ?>>> CONVERSIONS = ImmutableMap.<Class<?>, Map<Class<?>, Function<Object, ?>>>builder()
+        .put(Double.class, ImmutableMap.<Class<?>, Function<Object, ?>>builder()
+            .put(BigDecimal.class, arg -> ((BigDecimal) arg).doubleValue())
+            .put(Byte.class, arg -> ((Byte) arg).doubleValue())
+            .put(Double.class, arg -> arg)
+            .put(Float.class, arg -> ((Float) arg).doubleValue())
+            .put(Integer.class, arg -> ((Integer) arg).doubleValue())
+            .put(Long.class, arg -> ((Long) arg).doubleValue())
+            .put(Number.class, arg -> ((Number) arg).doubleValue())
+            .put(Short.class, arg -> ((Short) arg).doubleValue())
+            .put(String.class, arg -> Double.parseDouble((String)arg))
+            .build()
+        )
+        .put(Long.class, ImmutableMap.<Class<?>, Function<Object, ?>>builder()
+            .put(BigDecimal.class, arg -> ((BigDecimal) arg).longValue())
+            .put(Byte.class, arg -> ((Byte) arg).longValue())
+            .put(Double.class, arg -> ((Double) arg).longValue())
+            .put(Float.class, arg -> ((Float) arg).longValue())
+            .put(Integer.class, arg -> ((Integer) arg).longValue())
+            .put(Long.class, arg -> arg)
+            .put(Number.class, arg -> ((Number) arg).longValue())
+            .put(Short.class, arg -> ((Short) arg).longValue())
+            .put(String.class, arg -> Long.parseLong((String)arg))

Review Comment:
   why not `Double.parseDouble((String)arg).longValue()`?





> [Broker-J] Broker-J Query REST API improvements
> -----------------------------------------------
>
>                 Key: QPID-8581
>                 URL: https://issues.apache.org/jira/browse/QPID-8581
>             Project: Qpid
>          Issue Type: Improvement
>          Components: Broker-J
>    Affects Versions: qpid-java-broker-8.0.6
>            Reporter: Daniil Kirilyuk
>            Priority: Minor
>
> Existing broker REST API (including broker query API) doesn’t support such functions as aggregation, grouping and using nested logical operators, which could be useful for gathering broker statistics and reporting.
> We suggest to add a new POST method for the endpoint /api/latest/querybroker and supply search criteria using JSON body. (Existing GET method implementation should stay without changes for keeping backwards compatibility and because of difficulties of parsing complicated search conditions from URI parameters when using GET request method).
> The new functionality should add
>  * logical OR operator (absent in current API)
>  * aggregation operators AVG, CNT, MAX, MIN, SUM
>  * grouping aggregation
>  * new numeric / datetime / string functions



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org