[jira] [Commented] (PHOENIX-6687) The region server hosting the SYSTEM.CATALOG fails to serve any metadata requests as default handler pool threads are exhausted.

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17568216#comment-17568216 ] 

ASF GitHub Bot commented on PHOENIX-6687:
-----------------------------------------

gjacoby126 commented on code in PR #1440:
URL: https://github.com/apache/phoenix/pull/1440#discussion_r923869765


##########
phoenix-core/src/main/java/org/apache/hadoop/hbase/ipc/controller/ServerSideRPCControllerFactory.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.ipc.controller;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ipc.RpcControllerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link RpcControllerFactory} that should only be used when
+ * making server-server remote RPCs to the region servers hosting Phoenix SYSTEM tables.

Review Comment:
   OK, so let me make sure I understnad properly. 
   
   Direct operations on a system table (UPSERTs or queries) would use the MetadataRpcController and coproc calls (such as CREATE and ALTER operations where MetadataClient calls MetadataEndpointImpl operations) would use ServerSideRPCController? And there's no way to consolidate the two in either direction because of the mismatch between shaded and unshaded protobufs disallows casting? 



##########
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java:
##########
@@ -1788,6 +1799,50 @@ private MetaDataMutationResult childLinkMetaDataCoprocessorExec(byte[] parentTab
         }
     }
 
+    @VisibleForTesting
+    protected RpcController getController() {
+        return getController(SYSTEM_CATALOG_HBASE_TABLE_NAME);
+    }
+
+        /**
+         * If configured to use the metadata handler pool for server side connections,
+         * use the {@link org.apache.hadoop.hbase.ipc.controller.MetadataRpcController}

Review Comment:
   This comment appears to be incorrect (it says it's using the MetadataRpcController)





> The region server hosting the SYSTEM.CATALOG fails to serve any metadata requests as default handler pool  threads are exhausted.
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-6687
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-6687
>             Project: Phoenix
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 5.1.0, 5.1.1, 4.16.1, 5.2.0, 5.1.2
>            Reporter: Jacob Isaac
>            Assignee: Jacob Isaac
>            Priority: Major
>             Fix For: 4.17.0, 5.2.0
>
>         Attachments: stacktraces.txt
>
>
> When SYSTEM.CATALOG region server is restarted and the server is experiencing heavy metadata call volume.
> The stack traces indicate that all the default handler pool threads are waiting for the CQSI.init thread to finish initializing.
> The CQSI.init thread itself cannot proceed since it cannot complete the second RPC call (org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility) due to thread starvation.
> For e.g
> The following [code|https://github.com/apache/phoenix/blob/3cff97087d79b85e282fca4ac69ddf499fb1f40f/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L661] turned the getTable(..) into needing an additional server-to-server RPC call when initializing a PhoenixConnection (CQSI.init) for the first time on the JVM. 
> It is well-known that server-to-server RPC calls are prone to deadlocking due to thread pool exhaustion.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)