You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Wenzhe Zhou (Jira)" <ji...@apache.org> on 2022/10/18 19:00:00 UTC
[jira] [Comment Edited] (IMPALA-11674) Fix IsPeekTimeoutTException and IsReadTimeoutTException for thrift-0.16.0
[ https://issues.apache.org/jira/browse/IMPALA-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619734#comment-17619734 ]
Wenzhe Zhou edited comment on IMPALA-11674 at 10/18/22 6:59 PM:
----------------------------------------------------------------
TSSLSocket::peek() in TSSLSocket.cpp was changed to call TSSLSocket::waitForEvent(). When TSSLSocket::waitForEvent() call THRIFT_POLL (which is poll() function on Linux) with positive timeout value, THRIFT_POLL return 0 when the call is timed out (https://linux.die.net/man/2/poll). So that TSSLSocket::waitForEvent() throw exception TTransportException(TTransportException::TIMED_OUT, "THRIFT_POLL (timed out)") for timeout.
IsReadTimeoutTException() and IsPeekTimeoutTException() should be updated to check new type of exception. Otherwise the functions return wrong values for timeout, which cause TAcceptQueueServer::Peek() to rethrow the exception to caller TAcceptQueueServer::run(). TAcceptQueueServer::run() then will write log message "AcceptQueueServer client died: THRIFT_POLL (timed out)", and close the connection.
In one reported case, client thrift connections were closed after 30 seconds with lots of log message "AcceptQueueServer client died: THRIFT_POLL (timed out)" in coordinator log file. The behavior was matching above code analysis.
[~rizaon] Please verify if my code analysis make sense. I think we have same issue for Thrift 0.11.0.
cc: [~joemcdonnell]
was (Author: wzhou):
TSSLSocket::peek() in TSSLSocket.cpp was changed to TSSLSocket::waitForEvent(). When TSSLSocket::waitForEvent() call THRIFT_POLL (which is poll() function on Linux) with positive timeout value, THRIFT_POLL return 0 when the call is timed out (https://linux.die.net/man/2/poll). So that TSSLSocket::waitForEvent() throw exception TTransportException(TTransportException::TIMED_OUT, "THRIFT_POLL (timed out)") for timeout.
IsReadTimeoutTException() and IsPeekTimeoutTException() should be updated to check new type of exception. Otherwise the functions return wrong values for timeout, which cause TAcceptQueueServer::Peek() to rethrow the exception to caller TAcceptQueueServer::run(). TAcceptQueueServer::run() will write log message "AcceptQueueServer client died: THRIFT_POLL (timed out)", then close the connection.
In one reported case, client thrift connections were closed after 30 seconds with lots of log message "AcceptQueueServer client died: THRIFT_POLL (timed out)" in coordinator log file. The behavior was matching above code analysis.
[~rizaon] Please verify if my code analysis make sense. I think we have same issue for Thrift 0.11.0.
cc: [~joemcdonnell]
> Fix IsPeekTimeoutTException and IsReadTimeoutTException for thrift-0.16.0
> -------------------------------------------------------------------------
>
> Key: IMPALA-11674
> URL: https://issues.apache.org/jira/browse/IMPALA-11674
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.2.0
> Reporter: Wenzhe Zhou
> Assignee: Riza Suminto
> Priority: Major
>
> IMPALA-7825 upgraded Thrift version from 0.9.3 to 0.11.0, IMPALA-11384 upgraded CPP Thrift components from 0.11.0 to Thrift-0.16.0.
> Functions IsPeekTimeoutTException() and IsReadTimeoutTException() in be/src/rpc/thrift-util.cc make assumption about the implementation of read(), peek(), write() and write_partial() in TSocket.cpp and TSSLSocket.cpp. The functions read() and peek() in TSSLSocket.cpp were changed in version 0.11.0 and 0.16.0 to throw different exception for timeout. This cause IsPeekTimeoutTException() and IsReadTimeoutTException() return wrong value after upgrade thrift, which in turn cause TAcceptQueueServer::Peek() to rethrow the exception to caller TAcceptQueueServer::run() and make TAcceptQueueServer::run() to close the connection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org