You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (Jira)" <ji...@apache.org> on 2020/02/07 19:41:00 UTC
[jira] [Resolved] (IMPALA-9345) Hang in
TestOrc::test_type_conversions when running create table in hive
[ https://issues.apache.org/jira/browse/IMPALA-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-9345.
-----------------------------------
Fix Version/s: Impala 3.4.0
Resolution: Fixed
Fixed the test failure. IMPALA-9365 tracks making the infra fixes required to reenable it
> Hang in TestOrc::test_type_conversions when running create table in hive
> ------------------------------------------------------------------------
>
> Key: IMPALA-9345
> URL: https://issues.apache.org/jira/browse/IMPALA-9345
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: broken-build, flaky
> Fix For: Impala 3.4.0
>
> Attachments: hive-server2.log, hive-server2.log
>
>
> {noformat}
> query_test.test_scanners.TestOrc.test_type_conversions[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: orc/def/block] (from pytest)
> Failing for the past 1 build (Since Failed#93 )
> Took 2 hr 0 min.
> add description
> Error Message
> query_test/test_scanners.py:1337: in test_type_conversions self.run_test_case('DataErrorsTest/orc-type-checks', vector, unique_database) common/impala_test_suite.py:659: in run_test_case result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:610: in __exec_in_hive result = h.execute(query, user=user) common/impala_connection.py:334: in execute r = self.__fetch_results(handle, profile_format=profile_format) common/impala_connection.py:441: in __fetch_results cursor._wait_to_finish() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:405: in _wait_to_finish resp = self._last_operation._rpc('GetOperationStatus', req) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:992: in _rpc response = self._execute(func_name, request) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1009: in _execute return func(request) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:517: in GetOperationStatus return self.recv_GetOperationStatus() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:529: in recv_GetOperationStatus (fname, mtype, rseqid) = iprot.readMessageBegin() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:126: in readMessageBegin sz = self.readI32() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:206: in readI32 buff = self.trans.readAll(4) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll chunk = self.read(sz - have) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:159: in read self._read_frame() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:163: in _read_frame header = read_all_compat(self._trans, 4) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/six.py:31: in <lambda> read_all_compat = lambda trans, sz: trans.readAll(sz) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll chunk = self.read(sz - have) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py:105: in read buff = self.handle.recv(sz) E Failed: Timeout >7200s
> Stacktrace
> query_test/test_scanners.py:1337: in test_type_conversions
> self.run_test_case('DataErrorsTest/orc-type-checks', vector, unique_database)
> common/impala_test_suite.py:659: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:610: in __exec_in_hive
> result = h.execute(query, user=user)
> common/impala_connection.py:334: in execute
> r = self.__fetch_results(handle, profile_format=profile_format)
> common/impala_connection.py:441: in __fetch_results
> cursor._wait_to_finish()
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:405: in _wait_to_finish
> resp = self._last_operation._rpc('GetOperationStatus', req)
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:992: in _rpc
> response = self._execute(func_name, request)
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1009: in _execute
> return func(request)
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:517: in GetOperationStatus
> return self.recv_GetOperationStatus()
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:529: in recv_GetOperationStatus
> (fname, mtype, rseqid) = iprot.readMessageBegin()
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:126: in readMessageBegin
> sz = self.readI32()
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:206: in readI32
> buff = self.trans.readAll(4)
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll
> chunk = self.read(sz - have)
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:159: in read
> self._read_frame()
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:163: in _read_frame
> header = read_all_compat(self._trans, 4)
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/six.py:31: in <lambda>
> read_all_compat = lambda trans, sz: trans.readAll(sz)
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll
> chunk = self.read(sz - have)
> /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py:105: in read
> buff = self.handle.recv(sz)
> E Failed: Timeout >7200s
> {noformat}
> This has happened on multiple builds, all running against S3. It seems to be happening with high frequency on S3, but isn't deterministic - I saw this commit both succeed and fail:
> {noformat}
> IMPALA-9334: Log node blacklisting removals in ClusterMembershipMgr
> {noformat}
> All failures did occur after " IMPALA-9242: Bump CDH_BUILD_NUMBER to 1814051" but not sure if that's related or not.
> Sample hive log:
> {noformat}
> tarmstrong@tarmstrong-box:~/tmplogs/impala-asf-master-core-s3-data-cache-93/logs/cluster$ grep a6c81474e8c hive*
> grep: hive: Is a directory
> hive-server2.log:2020-01-28T11:10:49,558 INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] ql.Driver: Compiling command(queryId=jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9): create external table test_type_conversions_3e81315e.union_complextypes(
> hive-server2.log:2020-01-28T11:10:50,798 INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] ql.Driver: Completed compiling command(queryId=jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9); Time taken: 1.274 seconds
> hive-server2.log:2020-01-28T13:10:16,567 INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] operation.Operation: The running operation has been successfully interrupted: jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)