You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Michael Smith (Jira)" <ji...@apache.org> on 2022/10/18 23:03:00 UTC
[jira] [Comment Edited] (IMPALA-11671) run-all-tests.sh with Ozone fails listing DFS files
[ https://issues.apache.org/jira/browse/IMPALA-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619882#comment-17619882 ]
Michael Smith edited comment on IMPALA-11671 at 10/18/22 11:02 PM:
-------------------------------------------------------------------
Ok, not a timing issue. Seems related to HDDS-4974.
Basic repro is
{code}
export TARGET_FILESYSTEM=ozone FE_TEST=false BE_TEST=false CLUSTER_TEST=false JDBC_TEST=false EE_TEST_FILES="query_test/test_cancellation.py query_test/test_decimal_queries.py"
run-all-tests.sh
{code}
The same few files always show up
{code}
ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/test_insert_select_exprs_5162abf6.db/overflowed_decimal_tbl_2/_impala_insert_staging/33400fc93fdceeb6_ba5864fb00000000
ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/test_insert_select_exprs_54f8f95e.db/overflowed_decimal_tbl_2/_impala_insert_staging/6f4c7b07046fe82b_18c45c9000000000
ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/tpch.db/ctas_cancel/_impala_insert_staging/914d22352488f593_3bdb75fd00000000
ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/tpch.db/ctas_cancel1665414798172/_impala_insert_staging/9143acc325c766b8_e2699dc200000000
ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/tpch.db/ctas_cancel1665414840184/_impala_insert_staging/6d41be5290cde484_c74aef7f00000000
{code}
Only option I can think of for avoiding this is to skip the trash when cleaning up test fixtures.
was (Author: JIRAUSER288956):
Ok, not a timing issue. Seems related to HDDS-4974.
Basic repro is
{code}
export TARGET_FILESYSTEM=ozone FE_TEST=false BE_TEST=false CLUSTER_TEST=false JDBC_TEST=false EE_TEST_FILES=query_test/test_cancellation.py
run-all-tests.sh
{code}
The same few files always show up
{code}
ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/test_insert_select_exprs_5162abf6.db/overflowed_decimal_tbl_2/_impala_insert_staging/33400fc93fdceeb6_ba5864fb00000000
ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/test_insert_select_exprs_54f8f95e.db/overflowed_decimal_tbl_2/_impala_insert_staging/6f4c7b07046fe82b_18c45c9000000000
ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/tpch.db/ctas_cancel/_impala_insert_staging/914d22352488f593_3bdb75fd00000000
ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/tpch.db/ctas_cancel1665414798172/_impala_insert_staging/9143acc325c766b8_e2699dc200000000
ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/tpch.db/ctas_cancel1665414840184/_impala_insert_staging/6d41be5290cde484_c74aef7f00000000
{code}
Only option I can think of for avoiding this is to skip the trash when cleaning up test fixtures.
> run-all-tests.sh with Ozone fails listing DFS files
> ---------------------------------------------------
>
> Key: IMPALA-11671
> URL: https://issues.apache.org/jira/browse/IMPALA-11671
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.1.0
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Critical
>
> {code}
> hdfs dfs -ls -R ${FILESYSTEM_PREFIX}/test-warehouse \
> > ${IMPALA_LOGS_DIR}/file-list-end-${i}.log 2>&1
> {code}
> in run-all-tests.sh fails when testing with Ozone. Errors like
> {code}
> ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/test_insert_select_exprs_5162abf6.db/overflowed_decimal_tbl_2/_impala_insert_staging/c94353a630109577_202e902000000000
> ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/tpch.db/ctas_cancel/_impala_insert_staging/454c6614d6c20d17_94fac58c00000000
> ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/tpch.db/ctas_cancel1665933223846/_impala_insert_staging/7a46352d463f457e_fcabb6e000000000
> ls: Unable to get file status: volume: impala bucket: test-warehouse key: .Trash/jenkins/Current/impala/test-warehouse/tpch.db/ctas_cancel1665933261855/_impala_insert_staging/d24d1c35c2a9360c_90e32ce200000000
> {code}
> are visible in file-list-end-1.log. Current theory is that Ozone is cleaning up trash files while we list them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org