[jira] [Created] (DRILL-6187) Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on

["Sorabh Hamirwasia (JIRA)" <ji...@apache.org>]

Sorabh Hamirwasia created DRILL-6187:
----------------------------------------

             Summary: Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on
                 Key: DRILL-6187
                 URL: https://issues.apache.org/jira/browse/DRILL-6187
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - RPC, Security
            Reporter: Sorabh Hamirwasia
            Assignee: Sorabh Hamirwasia
             Fix For: 1.13.0


 
{color:#000000}Below is the summary of issue: {color}
 
{color:#000000}*Scenario:*{color}
{color:#000000}It seems like first sendRecordBatch was sent to Foreman which initiated the Authentication handshake. But before initiating handshake for auth we establish a connection and store that in a registry. Now if in parallel there is another recordBatch (by a different minor fragment running on same Drillbit) to be sent then that will see the connection available in registry and will initiate the send. Before the authentication is completed this second request reached foreman and it throws below exception saying RPC type 3 message is not allowed and closes the connection. This also fails the authentication handshake which was in progress.{color}{color:#000000} Here the logs with details:{color}
{color:#000000} {color}
{color:#000000}*Forman received the SASL_START message from another node:*{color}
{color:#000000}*_2018-02-21 18:43:30,759 [_*{color}{color:#000000}_BitServer-4] TRACE o.a.d.e.r.s.ServerAuthenticationHandler - Received SASL message SASL_START from /10.10.100.161:35482_{color}
{color:#000000} {color}
{color:#000000}*Then around same time it received another message from client of Rpc Type 3 which is for SendRecordBatch and fails since handshake is not completed yet.*{color}
{color:#000000} {color}
{color:#000000}*_2018-02-21 18:43:30,762_*{color}{color:#000000} _[BitServer-4] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /10.10.100.162:31012 <--> /__10.10.100.161:35482_ _(data server).  Closing connection._{color}
{color:#000000}_io.netty.handler.codec.DecoderException: org.apache.drill.exec.rpc.RpcException: Request of type 3 is not allowed without authentication. Client on /__10.10.100.161:35482_ _must authenticate before making requests. Connection dropped. [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0]_{color}
{color:#000000} {color}
{color:#000000}*Then client receives an channel closed exception:*{color}
{color:#000000} {color}
{color:#000000}*2018-02-21 18:43:30,764 [*{color}{color:#000000}BitClient-4] WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel.  Connection: /_10.10.100.161:35482_ <--> _10.10.100.162:31012_ (data client){color}
{color:#000000} {color}
{color:#000000}*and due to this it's initial command for authentication also fails. Since there is channel closed exception above I will think that triggered the failure of authentication request as well.*{color}
{color:#000000} {color}
{color:#000000}_Caused by: org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection.  Failure type AUTHENTICATION._{color}
{color:#000000}        _at org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67) ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
{color:#000000}        _at org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66) ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
{color:#000000}        _at org.apache.drill.exec.rpc.data.DataTunnel$SendBatchAsyncListen.connectionFailed(DataTunnel.java:166) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
{color:#000000}        _at org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:203) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
{color:#000000}        _at org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:147) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
{color:#000000}        _at org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.waitAndRun(ReconnectingConnection.java:122) ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
{color:#000000}        _at org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:83) ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
{color:#000000}        _at org.apache.drill.exec.rpc.data.DataTunnel._{color}{color:#000000}*_sendRecordBatch_*{color}{color:#000000}_(DataTunnel.java:84) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
{color:#000000}        _at org.apache.drill.exec.ops.AccountingDataTunnel.sendRecordBatch(AccountingDataTunnel.java:45) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
{color:#000000}        _at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:127) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
{color:#000000} {color}
{color:#000000}So I think there is a concurrency issue where even though the authentication is not completed the other requests are send to remote node as soon as TCP connection is available. Instead it should wait until authentication is completed. Something like TCP connection should be made available from registry only if authentication is completed.{color}
 
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)