You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/09/22 07:51:43 UTC
[GitHub] [lucene-solr] atris opened a new pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
atris opened a new pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906
This commit moves rate limiters configuration from web.xml to clusterprops.json and allows setting the configuration using a newly added command.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
Posted by GitBox <gi...@apache.org>.
noblepaul commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r494062845
##########
File path: solr/core/src/java/org/apache/solr/servlet/RateLimiterConfig.java
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.servlet;
Review comment:
I don't think it should belong here. only servlet filters etc are supposed to be there.
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
Review comment:
`clusterProperties.setClusterProperties()` is a ZK write op. We are doing 2 operations here. We must avoid it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
Posted by GitBox <gi...@apache.org>.
noblepaul commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r494062845
##########
File path: solr/core/src/java/org/apache/solr/servlet/RateLimiterConfig.java
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.servlet;
Review comment:
I don't think it should belong here. only servlet filters etc are supposed to be there.
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
Review comment:
`clusterProperties.setClusterProperties()` is a ZK write op. We are doing 2 operations here. We must avoid it
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -206,7 +209,7 @@ public void setObjProperty(PayloadObj<ClusterPropInfo> obj) {
public void setProperty(PayloadObj<Map<String,String>> obj) throws Exception {
Map<String,Object> m = obj.getDataMap();
m.put("action", CLUSTERPROP.toString());
- collectionsHandler.handleRequestBody(wrapParams(obj.getRequest(),m ), obj.getResponse());
+ collectionsHandler.handleRequestBody(wrapParams(obj.getRequest(), m), obj.getResponse());
Review comment:
why is there a change here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] atris commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r493238083
##########
File path: solr/core/src/java/org/apache/solr/servlet/RateLimiterConfig.java
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.servlet;
Review comment:
RequestRateLimiter lives within this package -- and since they directly and only work with SDF, should they not belong here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] atris commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r493214015
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
Review comment:
I believe that if we dont set it to null before writing, the new properties will be appended to existing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] atris commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r493207484
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
+ clusterProperties.setClusterProperties(
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, placementPluginConfig));
+ }
+
+ } catch (Exception e) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error in API", e);
+ }
+ }
+
+ @Command(name = "set-ratelimiters")
+ public void setRateLimiters(PayloadObj<Map<String, Object>> obj) {
+ Map<String, Object> rateLimiterConfig = obj.getDataMap();
+ final boolean unset = rateLimiterConfig.isEmpty();
Review comment:
I actually prefer {}, seems cleaner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
Posted by GitBox <gi...@apache.org>.
noblepaul commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r494114574
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -206,7 +209,7 @@ public void setObjProperty(PayloadObj<ClusterPropInfo> obj) {
public void setProperty(PayloadObj<Map<String,String>> obj) throws Exception {
Map<String,Object> m = obj.getDataMap();
m.put("action", CLUSTERPROP.toString());
- collectionsHandler.handleRequestBody(wrapParams(obj.getRequest(),m ), obj.getResponse());
+ collectionsHandler.handleRequestBody(wrapParams(obj.getRequest(), m), obj.getResponse());
Review comment:
why is there a change here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
Posted by GitBox <gi...@apache.org>.
noblepaul commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r492668757
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
+ clusterProperties.setClusterProperties(
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, placementPluginConfig));
+ }
+
+ } catch (Exception e) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error in API", e);
+ }
+ }
+
+ @Command(name = "set-ratelimiters")
+ public void setRateLimiters(PayloadObj<Map<String, Object>> obj) {
+ Map<String, Object> rateLimiterConfig = obj.getDataMap();
+ final boolean unset = rateLimiterConfig.isEmpty();
Review comment:
wouldn't you do `set-ratelimters : null` instead of `set-ratelimters : {}`
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
+ clusterProperties.setClusterProperties(
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, placementPluginConfig));
+ }
+
+ } catch (Exception e) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error in API", e);
+ }
+ }
+
+ @Command(name = "set-ratelimiters")
Review comment:
why plural? are there multiple rate limiters?
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
+ clusterProperties.setClusterProperties(
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, placementPluginConfig));
+ }
+
+ } catch (Exception e) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error in API", e);
+ }
+ }
+
+ @Command(name = "set-ratelimiters")
+ public void setRateLimiters(PayloadObj<Map<String, Object>> obj) {
Review comment:
Why not have a strongly typed java object? why are you using a loose `Map` object
##########
File path: solr/core/src/java/org/apache/solr/servlet/RateLimiterConfig.java
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.servlet;
Review comment:
why is this in the `o.a.s.servlet` package? It's not a servlet/filter
##########
File path: solr/solr-ref-guide/src/rate-limiters.adoc
##########
@@ -31,67 +31,44 @@ pronounced under high stress in production workloads. The current implementation
resources for indexing.
== Rate Limiter Configurations
-The default rate limiter is search rate limiter. Accordingly, it can be configured in `web.xml` under `initParams` for
-`SolrRequestFilter`.
-
-[source,xml]
-----
-<filter-name>SolrRequestFilter</filter-name>
-----
+The default rate limiter is search rate limiter. Accordingly, it can be configured using the following command:
+
+ curl -X POST -H 'Content-type:application/json' -d '{
+ "set-ratelimiters": {
+ "rlEnabled": "true",
Review comment:
why have the `rl` prefix everywhere? This is already used in the context of a ratelimiter, why use attributes like `rlGuaranteedSlots` just use `guaranteedSlots`
##########
File path: solr/core/src/java/org/apache/solr/servlet/QueryRateLimiter.java
##########
@@ -17,39 +17,82 @@
package org.apache.solr.servlet;
-import javax.servlet.FilterConfig;
+import java.util.Map;
import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.data.Stat;
-import static org.apache.solr.servlet.RateLimitManager.DEFAULT_CONCURRENT_REQUESTS;
-import static org.apache.solr.servlet.RateLimitManager.DEFAULT_SLOT_ACQUISITION_TIMEOUT_MS;
+import static org.apache.solr.servlet.RateLimiterConfig.RL_CONFIG_KEY;
/** Implementation of RequestRateLimiter specific to query request types. Most of the actual work is delegated
* to the parent class but specific configurations and parsing are handled by this class.
*/
public class QueryRateLimiter extends RequestRateLimiter {
- final static String IS_QUERY_RATE_LIMITER_ENABLED = "isQueryRateLimiterEnabled";
- final static String MAX_QUERY_REQUESTS = "maxQueryRequests";
- final static String QUERY_WAIT_FOR_SLOT_ALLOCATION_INMS = "queryWaitForSlotAllocationInMS";
- final static String QUERY_GUARANTEED_SLOTS = "queryGuaranteedSlots";
- final static String QUERY_ALLOW_SLOT_BORROWING = "queryAllowSlotBorrowing";
-
- public QueryRateLimiter(FilterConfig filterConfig) {
- super(constructQueryRateLimiterConfig(filterConfig));
+
+ public QueryRateLimiter(SolrZkClient solrZkClient) {
+ super(constructQueryRateLimiterConfig(solrZkClient));
+ }
+
+ @SuppressWarnings({"unchecked"})
+ public void processConfigChange(Map<String, Object> properties) {
+ RateLimiterConfig rateLimiterConfig = getRateLimiterConfig();
+ Map<String, Object> propertiesMap = (Map<String, Object>) properties.get(RL_CONFIG_KEY);
+
+ constructQueryRateLimiterConfigInternal(propertiesMap, rateLimiterConfig);
+ }
+
+ // To be used in initialization
+ @SuppressWarnings({"unchecked"})
+ private static RateLimiterConfig constructQueryRateLimiterConfig(SolrZkClient zkClient) {
+ try {
+
+ if (zkClient == null) {
+ return new RateLimiterConfig(SolrRequest.SolrRequestType.QUERY);
+ }
+
+ Map<String, Object> clusterPropsJson = (Map<String, Object>) Utils.fromJSON(zkClient.getData(ZkStateReader.CLUSTER_PROPS, null, new Stat(), true));
+ Map<String, Object> propertiesMap = (Map<String, Object>) clusterPropsJson.get(RL_CONFIG_KEY);
+ RateLimiterConfig rateLimiterConfig = new RateLimiterConfig(SolrRequest.SolrRequestType.QUERY);
+
+ constructQueryRateLimiterConfigInternal(propertiesMap, rateLimiterConfig);
+
+ return rateLimiterConfig;
+ } catch (KeeperException.NoNodeException e) {
+ return new RateLimiterConfig(SolrRequest.SolrRequestType.QUERY);
+ } catch (KeeperException | InterruptedException e) {
+ throw new RuntimeException("Error reading cluster property", SolrZkClient.checkInterrupted(e));
+ }
}
- protected static RequestRateLimiter.RateLimiterConfig constructQueryRateLimiterConfig(FilterConfig filterConfig) {
- RequestRateLimiter.RateLimiterConfig queryRateLimiterConfig = new RequestRateLimiter.RateLimiterConfig();
+ private static void constructQueryRateLimiterConfigInternal(Map<String, Object> propertiesMap, RateLimiterConfig rateLimiterConfig) {
Review comment:
Why are you not using a POJO and jackson deserialization, like it's done elsewhere? why all these boilerplate code
##########
File path: solr/solr-ref-guide/src/rate-limiters.adoc
##########
@@ -31,67 +31,44 @@ pronounced under high stress in production workloads. The current implementation
resources for indexing.
== Rate Limiter Configurations
-The default rate limiter is search rate limiter. Accordingly, it can be configured in `web.xml` under `initParams` for
-`SolrRequestFilter`.
-
-[source,xml]
-----
-<filter-name>SolrRequestFilter</filter-name>
-----
+The default rate limiter is search rate limiter. Accordingly, it can be configured using the following command:
+
+ curl -X POST -H 'Content-type:application/json' -d '{
+ "set-ratelimiters": {
+ "rlEnabled": "true",
Review comment:
why string `"true"` why not `true` ?
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
Review comment:
why 2 write ops? Only one should be enough
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
Posted by GitBox <gi...@apache.org>.
noblepaul commented on a change in pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906#discussion_r492668757
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
+ clusterProperties.setClusterProperties(
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, placementPluginConfig));
+ }
+
+ } catch (Exception e) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error in API", e);
+ }
+ }
+
+ @Command(name = "set-ratelimiters")
+ public void setRateLimiters(PayloadObj<Map<String, Object>> obj) {
+ Map<String, Object> rateLimiterConfig = obj.getDataMap();
+ final boolean unset = rateLimiterConfig.isEmpty();
Review comment:
wouldn't you do `set-ratelimters : null` instead of `set-ratelimters : {}`
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
+ clusterProperties.setClusterProperties(
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, placementPluginConfig));
+ }
+
+ } catch (Exception e) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error in API", e);
+ }
+ }
+
+ @Command(name = "set-ratelimiters")
Review comment:
why plural? are there multiple rate limiters?
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
+ clusterProperties.setClusterProperties(
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, placementPluginConfig));
+ }
+
+ } catch (Exception e) {
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error in API", e);
+ }
+ }
+
+ @Command(name = "set-ratelimiters")
+ public void setRateLimiters(PayloadObj<Map<String, Object>> obj) {
Review comment:
Why not have a strongly typed java object? why are you using a loose `Map` object
##########
File path: solr/core/src/java/org/apache/solr/servlet/RateLimiterConfig.java
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.servlet;
Review comment:
why is this in the `o.a.s.servlet` package? It's not a servlet/filter
##########
File path: solr/solr-ref-guide/src/rate-limiters.adoc
##########
@@ -31,67 +31,44 @@ pronounced under high stress in production workloads. The current implementation
resources for indexing.
== Rate Limiter Configurations
-The default rate limiter is search rate limiter. Accordingly, it can be configured in `web.xml` under `initParams` for
-`SolrRequestFilter`.
-
-[source,xml]
-----
-<filter-name>SolrRequestFilter</filter-name>
-----
+The default rate limiter is search rate limiter. Accordingly, it can be configured using the following command:
+
+ curl -X POST -H 'Content-type:application/json' -d '{
+ "set-ratelimiters": {
+ "rlEnabled": "true",
Review comment:
why have the `rl` prefix everywhere? This is already used in the context of a ratelimiter, why use attributes like `rlGuaranteedSlots` just use `guaranteedSlots`
##########
File path: solr/core/src/java/org/apache/solr/servlet/QueryRateLimiter.java
##########
@@ -17,39 +17,82 @@
package org.apache.solr.servlet;
-import javax.servlet.FilterConfig;
+import java.util.Map;
import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.common.util.Utils;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.data.Stat;
-import static org.apache.solr.servlet.RateLimitManager.DEFAULT_CONCURRENT_REQUESTS;
-import static org.apache.solr.servlet.RateLimitManager.DEFAULT_SLOT_ACQUISITION_TIMEOUT_MS;
+import static org.apache.solr.servlet.RateLimiterConfig.RL_CONFIG_KEY;
/** Implementation of RequestRateLimiter specific to query request types. Most of the actual work is delegated
* to the parent class but specific configurations and parsing are handled by this class.
*/
public class QueryRateLimiter extends RequestRateLimiter {
- final static String IS_QUERY_RATE_LIMITER_ENABLED = "isQueryRateLimiterEnabled";
- final static String MAX_QUERY_REQUESTS = "maxQueryRequests";
- final static String QUERY_WAIT_FOR_SLOT_ALLOCATION_INMS = "queryWaitForSlotAllocationInMS";
- final static String QUERY_GUARANTEED_SLOTS = "queryGuaranteedSlots";
- final static String QUERY_ALLOW_SLOT_BORROWING = "queryAllowSlotBorrowing";
-
- public QueryRateLimiter(FilterConfig filterConfig) {
- super(constructQueryRateLimiterConfig(filterConfig));
+
+ public QueryRateLimiter(SolrZkClient solrZkClient) {
+ super(constructQueryRateLimiterConfig(solrZkClient));
+ }
+
+ @SuppressWarnings({"unchecked"})
+ public void processConfigChange(Map<String, Object> properties) {
+ RateLimiterConfig rateLimiterConfig = getRateLimiterConfig();
+ Map<String, Object> propertiesMap = (Map<String, Object>) properties.get(RL_CONFIG_KEY);
+
+ constructQueryRateLimiterConfigInternal(propertiesMap, rateLimiterConfig);
+ }
+
+ // To be used in initialization
+ @SuppressWarnings({"unchecked"})
+ private static RateLimiterConfig constructQueryRateLimiterConfig(SolrZkClient zkClient) {
+ try {
+
+ if (zkClient == null) {
+ return new RateLimiterConfig(SolrRequest.SolrRequestType.QUERY);
+ }
+
+ Map<String, Object> clusterPropsJson = (Map<String, Object>) Utils.fromJSON(zkClient.getData(ZkStateReader.CLUSTER_PROPS, null, new Stat(), true));
+ Map<String, Object> propertiesMap = (Map<String, Object>) clusterPropsJson.get(RL_CONFIG_KEY);
+ RateLimiterConfig rateLimiterConfig = new RateLimiterConfig(SolrRequest.SolrRequestType.QUERY);
+
+ constructQueryRateLimiterConfigInternal(propertiesMap, rateLimiterConfig);
+
+ return rateLimiterConfig;
+ } catch (KeeperException.NoNodeException e) {
+ return new RateLimiterConfig(SolrRequest.SolrRequestType.QUERY);
+ } catch (KeeperException | InterruptedException e) {
+ throw new RuntimeException("Error reading cluster property", SolrZkClient.checkInterrupted(e));
+ }
}
- protected static RequestRateLimiter.RateLimiterConfig constructQueryRateLimiterConfig(FilterConfig filterConfig) {
- RequestRateLimiter.RateLimiterConfig queryRateLimiterConfig = new RequestRateLimiter.RateLimiterConfig();
+ private static void constructQueryRateLimiterConfigInternal(Map<String, Object> propertiesMap, RateLimiterConfig rateLimiterConfig) {
Review comment:
Why are you not using a POJO and jackson deserialization, like it's done elsewhere? why all these boilerplate code
##########
File path: solr/solr-ref-guide/src/rate-limiters.adoc
##########
@@ -31,67 +31,44 @@ pronounced under high stress in production workloads. The current implementation
resources for indexing.
== Rate Limiter Configurations
-The default rate limiter is search rate limiter. Accordingly, it can be configured in `web.xml` under `initParams` for
-`SolrRequestFilter`.
-
-[source,xml]
-----
-<filter-name>SolrRequestFilter</filter-name>
-----
+The default rate limiter is search rate limiter. Accordingly, it can be configured using the following command:
+
+ curl -X POST -H 'Content-type:application/json' -d '{
+ "set-ratelimiters": {
+ "rlEnabled": "true",
Review comment:
why string `"true"` why not `true` ?
##########
File path: solr/core/src/java/org/apache/solr/handler/ClusterAPI.java
##########
@@ -171,10 +172,30 @@ public void setPlacementPlugin(PayloadObj<Map<String, Object>> obj) {
// Need to reset to null first otherwise the mappings in placementPluginConfig are added to existing ones
// in /clusterprops.json rather than replacing them. If removing the config, that's all we do.
clusterProperties.setClusterProperties(
- Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ Collections.singletonMap(PlacementPluginConfigImpl.PLACEMENT_PLUGIN_CONFIG_KEY, null));
+ if (!unset) {
Review comment:
why 2 write ops? Only one should be enough
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] atris merged pull request #1906: SOLR-13528: Implement API Based Config For Rate Limiters
Posted by GitBox <gi...@apache.org>.
atris merged pull request #1906:
URL: https://github.com/apache/lucene-solr/pull/1906
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org