You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2020/08/13 02:14:00 UTC
[jira] [Commented] (IMPALA-10077)
test_concurrent_invalidate_metadata timed out
[ https://issues.apache.org/jira/browse/IMPALA-10077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17176714#comment-17176714 ]
Quanlong Huang commented on IMPALA-10077:
-----------------------------------------
Found some unrelated exceptions in catalogd's log:
{code}
I0812 01:48:31.159904 5375 CatalogServiceCatalog.java:1601] Invalidating all metadata. Version: 0
E0812 01:48:31.164568 5375 CatalogServiceCatalog.java:519] Error loading cache pools:
Java exception follows:
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:492)
at org.apache.impala.common.FileSystemUtil.getDistributedFileSystem(FileSystemUtil.java:465)
at org.apache.impala.catalog.CatalogServiceCatalog$CachePoolReader.run(CatalogServiceCatalog.java:512)
at org.apache.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:1620)
at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:137)
{code}
We should skip reading hdfs cache pools if the default filesystem is not a DistributedFileSystem. Will fix this as well in the patch.
> test_concurrent_invalidate_metadata timed out
> ---------------------------------------------
>
> Key: IMPALA-10077
> URL: https://issues.apache.org/jira/browse/IMPALA-10077
> Project: IMPALA
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Assignee: Quanlong Huang
> Priority: Major
> Labels: broken-build
>
> Encountered this in an S3 build.
> *Error Message*
> Failed: Timeout >120.0s
> *Stacktrace*
> custom_cluster/test_concurrent_ddls.py:191: in test_concurrent_invalidate_metadata
> r1.get(timeout=60)
> /usr/lib64/python2.7/multiprocessing/pool.py:548: in get
> self.wait(timeout)
> /usr/lib64/python2.7/multiprocessing/pool.py:543: in wait
> self._cond.wait(timeout)
> /usr/lib64/python2.7/threading.py:362: in wait
> _sleep(delay)
> E Failed: Timeout >120.0s
> *Standard Error*
> 01:48:26 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> 01:48:26 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 01:48:26 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 01:48:27 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 01:48:27 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 01:48:27 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 01:48:30 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 01:48:30 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 01:48:30 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
> 01:48:30 MainThread: Debug webpage not yet available: ('Connection aborted.', error(111, 'Connection refused'))
> 01:48:32 MainThread: Debug webpage did not become available in expected time.
> 01:48:32 MainThread: Waiting for num_known_live_backends=3. Current value: None
> 01:48:33 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 01:48:33 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
> 01:48:33 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 01:48:34 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 01:48:34 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
> 01:48:34 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 01:48:35 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 01:48:35 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
> 01:48:35 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 01:48:36 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 01:48:36 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
> 01:48:36 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 01:48:37 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 01:48:37 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
> 01:48:37 MainThread: num_known_live_backends has reached value: 3
> 01:48:37 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 01:48:37 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25001
> 01:48:37 MainThread: num_known_live_backends has reached value: 3
> 01:48:38 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 01:48:38 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25002
> 01:48:38 MainThread: num_known_live_backends has reached value: 3
> 01:48:38 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 executors).
> DEBUG:impala_cluster:Found 3 impalad/1 statestored/1 catalogd process(es)
> INFO:impala_service:Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25010
> INFO:impala_service:Metric 'statestore.live-backends' has reached desired value: 4
> DEBUG:impala_service:Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
> INFO:impala_service:num_known_live_backends has reached value: 3
> DEBUG:impala_service:Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25001
> INFO:impala_service:num_known_live_backends has reached value: 3
> DEBUG:impala_service:Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25002
> INFO:impala_service:num_known_live_backends has reached value: 3
> +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
> ~~~~~~~~~~~~~~~~~~~~ Stack of Thread-132 (140478892721920) ~~~~~~~~~~~~~~~~~~~~~
> File "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
> self.__bootstrap_inner()
> File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
> self.run()
> File "/usr/lib64/python2.7/threading.py", line 765, in run
> self.__target(*self.__args, **self.__kwargs)
> File "/usr/lib64/python2.7/multiprocessing/pool.py", line 376, in _handle_results
> task = get()
> File "/usr/lib64/python2.7/Queue.py", line 168, in get
> self.not_empty.wait()
> File "/usr/lib64/python2.7/threading.py", line 339, in wait
> waiter.acquire()
> ~~~~~~~~~~~~~~~~~~~~ Stack of Thread-131 (140478901114624) ~~~~~~~~~~~~~~~~~~~~~
> File "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
> self.__bootstrap_inner()
> File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
> self.run()
> File "/usr/lib64/python2.7/threading.py", line 765, in run
> self.__target(*self.__args, **self.__kwargs)
> File "/usr/lib64/python2.7/multiprocessing/pool.py", line 335, in _handle_tasks
> for taskseq, set_length in iter(taskqueue.get, None):
> File "/usr/lib64/python2.7/Queue.py", line 168, in get
> self.not_empty.wait()
> File "/usr/lib64/python2.7/threading.py", line 339, in wait
> waiter.acquire()
> ~~~~~~~~~~~~~~~~~~~~~ Stack of Thread-80 (140482340452096) ~~~~~~~~~~~~~~~~~~~~~
> File "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
> self.__bootstrap_inner()
> File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
> self.run()
> File "/usr/lib64/python2.7/threading.py", line 765, in run
> self.__target(*self.__args, **self.__kwargs)
> File "/usr/lib64/python2.7/multiprocessing/pool.py", line 102, in worker
> task = get()
> File "/usr/lib64/python2.7/Queue.py", line 168, in get
> self.not_empty.wait()
> File "/usr/lib64/python2.7/threading.py", line 339, in wait
> waiter.acquire()
> ~~~~~~~~~~~~~~~~~~~~ Stack of Thread-129 (140478917900032) ~~~~~~~~~~~~~~~~~~~~~
> File "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
> self.__bootstrap_inner()
> File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
> self.run()
> File "/usr/lib64/python2.7/threading.py", line 765, in run
> self.__target(*self.__args, **self.__kwargs)
> File "/usr/lib64/python2.7/multiprocessing/pool.py", line 113, in worker
> result = (True, func(*args, **kwds))
> File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/custom_cluster/test_concurrent_ddls.py", line 182, in run_invalidate_metadata
> self.execute_query_expect_success(tls.client, "invalidate metadata")
> File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/common/impala_test_suite.py", line 811, in wrapper
> return function(*args, **kwargs)
> File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/common/impala_test_suite.py", line 819, in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/common/impala_test_suite.py", line 909, in __execute_query
> return impalad_client.execute(query, user=user)
> File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/common/impala_connection.py", line 205, in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py", line 187, in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py", line 363, in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py", line 357, in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py", line 518, in __do_rpc
> return rpc()
> File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py", line 357, in <lambda>
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> File "/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/shell/gen-py/beeswaxd/BeeswaxService.py", line 143, in query
> return self.recv_query()
> File "/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/shell/gen-py/beeswaxd/BeeswaxService.py", line 155, in recv_query
> (fname, mtype, rseqid) = iprot.readMessageBegin()
> File "/data/jenkins/workspace/impala-cdpd-master-core-s3/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
> sz = self.readI32()
> File "/data/jenkins/workspace/impala-cdpd-master-core-s3/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 206, in readI32
> buff = self.trans.readAll(4)
> File "/data/jenkins/workspace/impala-cdpd-master-core-s3/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 58, in readAll
> chunk = self.read(sz - have)
> File "/data/jenkins/workspace/impala-cdpd-master-core-s3/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 159, in read
> self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
> File "/data/jenkins/workspace/impala-cdpd-master-core-s3/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/python/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 105, in read
> buff = self.handle.recv(sz)
> ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org