You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/12/03 11:59:00 UTC
[jira] [Commented] (IMPALA-10301) Insert query hangs in
test_local_catalog_ddls_with_invalidate_metadata_sync_ddl
[ https://issues.apache.org/jira/browse/IMPALA-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243144#comment-17243144 ]
ASF subversion and git services commented on IMPALA-10301:
----------------------------------------------------------
Commit 90944d7340337a75929466b590bea23fdb302337 in impala's branch refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=90944d7 ]
IMPALA-10369: Dump server stacktraces when test_concurrent_ddls.py timeout
Recently, we see many timeout failures of test_concurrent_ddls.py in S3
builds, e.g. IMPALA-10280, IMPALA-10301, IMPALA-10363. It'd be helpful
to dump the server stacktraces so we can understand why some RPCs are
slow/stuck.
This patch extracts the logic of dumping stacktraces in
script-timeout-check.sh to a separate script, dump-stacktraces.sh.
The script also dumps jstacks of HMS and NameNode. Dumping all these
stacktraces is time-consuming so we do them in parallel, which also
helps to get consistent snapshots of all servers.
When any tests in test_concurrent_ddls.py timeout, we use
dump-stacktraces.sh to dump the stacktraces before exit. Previously,
some tests depend on pytest.mark.timeout for detecting timeouts. It's
hard to add a customized callback for dumping server stacktraces. So
this patch refactors test_concurrent_ddls.py to only use timeout of
multiprocessing.
Tests:
- Tested the scripts locally.
- Verified the error handling of timeout logics in Jenkins jobs
Change-Id: I514cf2d0ff842805c0abf7211f2a395151174173
Reviewed-on: http://gerrit.cloudera.org:8080/16800
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Insert query hangs in test_local_catalog_ddls_with_invalidate_metadata_sync_ddl
> -------------------------------------------------------------------------------
>
> Key: IMPALA-10301
> URL: https://issues.apache.org/jira/browse/IMPALA-10301
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Qifan Chen
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: broken-build, flaky, hang
> Attachments: failure-output.txt, test_mixed_catalog_ddls_with_invalidate_metadata.tar.gz
>
>
> In impala-cdpd-master-staging-core-s3 test, the following error was seen (insert into a partitioned table timeout).
> Error Message
> {code:java}
> AssertionError: Query timeout(60s): insert overwrite table test_local_catalog_ddls_with_invalidate_metadata_sync_ddl_b87f02d6.test_2_part partition(j=2) values (1), (2), (3), (4), (5) assert False
> Stacktrace
> {code}
> {code:java}
> custom_cluster/test_concurrent_ddls.py:83: in test_local_catalog_ddls_with_invalidate_metadata_sync_ddl
> self._run_ddls_with_invalidation(unique_database, sync_ddl=True)
> custom_cluster/test_concurrent_ddls.py:146: in _run_ddls_with_invalidation
> for i in pool.imap_unordered(run_ddls, xrange(1, NUM_ITERS + 1)):
> /usr/lib64/python2.7/multiprocessing/pool.py:655: in next
> raise value
> E AssertionError: Query timeout(60s): insert overwrite table test_local_catalog_ddls_with_invalidate_metadata_sync_ddl_b87f02d6.test_2_part partition(j=2) values (1), (2), (3), (4), (5)
> E assert False
> {code}
> The URL is https://master-02.jenkins.cloudera.com/view/Impala/view/Evergreen-cdpd-master-staging/job/impala-cdpd-master-staging-core-s3/lastCompletedBuild/testReport/custom_cluster.test_concurrent_ddls/TestConcurrentDdls/test_local_catalog_ddls_with_invalidate_metadata_sync_ddl/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org