[jira] [Comment Edited] (PHOENIX-3360) Secondary index configuration is wrong

["William Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851023#comment-15851023 ] 

William Yang edited comment on PHOENIX-3360 at 2/3/17 4:08 AM:
---------------------------------------------------------------

I don't think that current patch has solved the problem because it is still setting the RPC class in RegionServer level instead of region level. 

We've encountered this problem too, our resolution is simple. The only thing we need to do is to set specified priority ONLY for index updates that originated from RS. We can do this in IndexWriterUtils#getDefaultDelegateHTableFactory() and CoprocessorHTableFactory. Method getDefaultDelegateHTableFactory() creates an htable factory to get all HTableInterface objects to do index updates in current region server. If we can make these HTableInterface objects have this RPC config, then all index update requests will have the right priority and will not interfere with other RS-RS requests.

Right now, IndexWriterUtils#getDefaultDelegateHTableFactory() is called at each data table regions init, then each index region has its own CoprocessorHTableFactory object. We need to do :
  * create a global CoprocessorHTableFactory object in getDefaultDelegateHTableFactory(), thus all data table regions will share the same CoprocessorHTableFactory.
  * in CoprocessorHTableFactory's ctor, create an HBase connection configured with our RPC controller class.
  * in CoprocessorHTableFactory#getTable(), get the table from the connection created in ctor instead of calling CoprocessorEnvironment#getTable() 

The only target here is to make sure all HTableInterface objects for updating index tables are created from the specified connection. I've done this and it works fine.




was (Author: yhxx511):
I don't think that current patch has solved the problem because it is still setting the RPC class in RegionServer level instead of region level. 

We've encountered this problem too, our resolution is simple. The only thing we need to do is to set specified priority ONLY for index updates that originated from RS. We can do this in IndexWriterUtils#getDefaultDelegateHTableFactory() and CoprocessorHTableFactory. Method getDefaultDelegateHTableFactory() creates an htable factory to get all HTableInterface objects to do index updates in current region server. If we can make these HTableInterface objects have this RPC config, then all index update requests will have the right priority and will not interfere with other RS-RS requests.

Right now, IndexWriterUtils#getDefaultDelegateHTableFactory() is called at each data table regions init, then each index region has its own CoprocessorHTableFactory object. We need to do :
  * create a single HBase connection for current RS and this connection is configured with this RPC class
  * all data table regions get HTableInterface from this connection instead of calling CoprocessorEnvironment#getTable() before doing a batch mutations to their index regions.

You can create this connection in getDefaultDelegateHTableFactory() or just let all regions share the same CoprocessorHTableFactory object. No matter which way you choose, the only target here is to make sure all HTableInterface objects for updating index tables are created from the specified connection. In this case, CoprocessorHTableFactory is no longer a suitable name as we don't get table from coprocessor env, but from an explicit connection object. 

I've done this and it works fine.



> Secondary index configuration is wrong
> --------------------------------------
>
>                 Key: PHOENIX-3360
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3360
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Enis Soztutar
>            Assignee: Rajeshbabu Chintaguntla
>            Priority: Critical
>             Fix For: 4.10.0
>
>         Attachments: PHOENIX-3360.patch
>
>
> IndexRpcScheduler allocates some handler threads and uses a higher priority for RPCs. The corresponding IndexRpcController is not used by default as it is, but used through ServerRpcControllerFactory that we configure from Ambari by default which sets the priority of the outgoing RPCs to either metadata priority, or the index priority.
> However, after reading code of IndexRpcController / ServerRpcController it seems that the IndexRPCController DOES NOT look at whether the outgoing RPC is for an Index table or not. It just sets ALL rpc priorities to be the index priority. The intention seems to be the case that ONLY on servers, we configure ServerRpcControllerFactory, and with clients we NEVER configure ServerRpcControllerFactory, but instead use ClientRpcControllerFactory. We configure ServerRpcControllerFactory from Ambari, which in affect makes it so that ALL rpcs from Phoenix are only handled by the index handlers by default. It means all deadlock cases are still there. 
> The documentation in https://phoenix.apache.org/secondary_indexing.html is also wrong in this sense. It does not talk about server side / client side. Plus this way of configuring different values is not how HBase configuration is deployed. We cannot have the configuration show the ServerRpcControllerFactory even only for server nodes, because the clients running on those nodes will also see the wrong values. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)