You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (Jira)" <ji...@apache.org> on 2020/01/30 20:12:00 UTC
[jira] [Created] (IMPALA-9345) Hang in
TestOrc::test_type_conversions when running create table in hive
Tim Armstrong created IMPALA-9345:
-------------------------------------
Summary: Hang in TestOrc::test_type_conversions when running create table in hive
Key: IMPALA-9345
URL: https://issues.apache.org/jira/browse/IMPALA-9345
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Reporter: Tim Armstrong
{noformat}
query_test.test_scanners.TestOrc.test_type_conversions[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: orc/def/block] (from pytest)
Failing for the past 1 build (Since Failed#93 )
Took 2 hr 0 min.
add description
Error Message
query_test/test_scanners.py:1337: in test_type_conversions self.run_test_case('DataErrorsTest/orc-type-checks', vector, unique_database) common/impala_test_suite.py:659: in run_test_case result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:610: in __exec_in_hive result = h.execute(query, user=user) common/impala_connection.py:334: in execute r = self.__fetch_results(handle, profile_format=profile_format) common/impala_connection.py:441: in __fetch_results cursor._wait_to_finish() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:405: in _wait_to_finish resp = self._last_operation._rpc('GetOperationStatus', req) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:992: in _rpc response = self._execute(func_name, request) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1009: in _execute return func(request) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:517: in GetOperationStatus return self.recv_GetOperationStatus() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:529: in recv_GetOperationStatus (fname, mtype, rseqid) = iprot.readMessageBegin() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:126: in readMessageBegin sz = self.readI32() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:206: in readI32 buff = self.trans.readAll(4) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll chunk = self.read(sz - have) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:159: in read self._read_frame() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:163: in _read_frame header = read_all_compat(self._trans, 4) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/six.py:31: in <lambda> read_all_compat = lambda trans, sz: trans.readAll(sz) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll chunk = self.read(sz - have) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py:105: in read buff = self.handle.recv(sz) E Failed: Timeout >7200s
Stacktrace
query_test/test_scanners.py:1337: in test_type_conversions
self.run_test_case('DataErrorsTest/orc-type-checks', vector, unique_database)
common/impala_test_suite.py:659: in run_test_case
result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
common/impala_test_suite.py:610: in __exec_in_hive
result = h.execute(query, user=user)
common/impala_connection.py:334: in execute
r = self.__fetch_results(handle, profile_format=profile_format)
common/impala_connection.py:441: in __fetch_results
cursor._wait_to_finish()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:405: in _wait_to_finish
resp = self._last_operation._rpc('GetOperationStatus', req)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:992: in _rpc
response = self._execute(func_name, request)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1009: in _execute
return func(request)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:517: in GetOperationStatus
return self.recv_GetOperationStatus()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:529: in recv_GetOperationStatus
(fname, mtype, rseqid) = iprot.readMessageBegin()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:126: in readMessageBegin
sz = self.readI32()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:206: in readI32
buff = self.trans.readAll(4)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll
chunk = self.read(sz - have)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:159: in read
self._read_frame()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:163: in _read_frame
header = read_all_compat(self._trans, 4)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/six.py:31: in <lambda>
read_all_compat = lambda trans, sz: trans.readAll(sz)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll
chunk = self.read(sz - have)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py:105: in read
buff = self.handle.recv(sz)
E Failed: Timeout >7200s
{noformat}
This has happened on multiple builds, all running against S3. It seems to be happening with high frequency on S3, but isn't deterministic - I saw this commit both succeeed and fail:
{noformat}
IMPALA-9334: Log node blacklisting removals in ClusterMembershipMgr
{noformat}
Sample hive log:
{noformat}
tarmstrong@tarmstrong-box:~/tmplogs/impala-asf-master-core-s3-data-cache-93/logs/cluster$ grep a6c81474e8c hive*
grep: hive: Is a directory
hive-server2.log:2020-01-28T11:10:49,558 INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] ql.Driver: Compiling command(queryId=jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9): create external table test_type_conversions_3e81315e.union_complextypes(
hive-server2.log:2020-01-28T11:10:50,798 INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] ql.Driver: Completed compiling command(queryId=jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9); Time taken: 1.274 seconds
hive-server2.log:2020-01-28T13:10:16,567 INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] operation.Operation: The running operation has been successfully interrupted: jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)