You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/03/20 05:27:00 UTC
[jira] [Commented] (IMPALA-9492)
TestRecoverPartitions::test_unescaped_string_partition failing on S3
[ https://issues.apache.org/jira/browse/IMPALA-9492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17063096#comment-17063096 ]
ASF subversion and git services commented on IMPALA-9492:
---------------------------------------------------------
Commit 030c12ab2c995a8618dcf601296e310203f70f8c in impala's branch refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=030c12a ]
IMPALA-9492: Fix test_unescaped_string_partition failing on S3
test_unescaped_string_partition in metadata/test_recover_partitions.py
use hdfs clients to create four partition directories with special
characters, i.e. single quote, double quotes and back slash. It aims to
test on whether ALTER TABLE RECOVER PARTITIONS can recognize those
directories correctly. However, when running against s3, only two
directories are created as expected, which causes the failure.
The reason is that when running against s3, we use hadoop cli for
operations. A shell command will be launched for each operation. Passing
arguments through shell results in duplicate unescaping. So the 4 dirs,
[p=', p=", p=\', p=\"] finally became [p=', p=", p=', p="], resulting in
two distinct directories. When the test running against hdfs, we use
webhdfs_client so don't have this issue.
Actually, we shouldn't use special characters in partition paths. Hive
converts them to their ascii hex values when creating partition
directories. E.g. partition paths of [p=', p=", p=\', p=\"] are
[p=%27, p=%22, p=%5C%27, p=%5C%22]. We should follow this rule when
creating directories in test. Also we won't have the above shell issue
on s3 anymore.
Tests:
- Added two more special partitions in test_unescaped_string_partition.
- Ran test_unescaped_string_partition in S3.
Change-Id: I63d149c9bdec52c2e1c0b25c8c3f0448cf7bdadb
Reviewed-on: http://gerrit.cloudera.org:8080/15475
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> TestRecoverPartitions::test_unescaped_string_partition failing on S3
> --------------------------------------------------------------------
>
> Key: IMPALA-9492
> URL: https://issues.apache.org/jira/browse/IMPALA-9492
> Project: IMPALA
> Issue Type: Bug
> Reporter: David Rorke
> Assignee: Quanlong Huang
> Priority: Blocker
> Labels: build-failure
>
> TestRecoverPartitions::test_unescaped_string_partition is a new test recently added by [https://gerrit.cloudera.org/#/c/15278/]
> It's failing when run against S3:
> {noformat}
> metadata/test_recover_partitions.py:374: in test_unescaped_string_partition
> assert self.count_partition(result.data) == 4
> E assert 2 == 4
> E + where 2 = <bound method TestRecoverPartitions.count_partition of <test_recover_partitions.TestRecoverPartitions object at 0x725e650>>(['"\t-1\t0\t0B\tNOT CACHED\tNOT CACHED\tTEXT\tfalse\ts3a://impala-test-uswest2-1/test-warehouse/test_unescaped_string_...ouse/test_unescaped_string_partition_2265687.db/test_unescaped_string_partition/p=%27", 'Total\t-1\t0\t0B\t0B\t\t\t\t'])
> E + where <bound method TestRecoverPartitions.count_partition of <test_recover_partitions.TestRecoverPartitions object at 0x725e650>> = <test_recover_partitions.TestRecoverPartitions object at 0x725e650>.count_partition
> E + and ['"\t-1\t0\t0B\tNOT CACHED\tNOT CACHED\tTEXT\tfalse\ts3a://impala-test-uswest2-1/test-warehouse/test_unescaped_string_...ouse/test_unescaped_string_partition_2265687.db/test_unescaped_string_partition/p=%27", 'Total\t-1\t0\t0B\t0B\t\t\t\t'] = <tests.beeswax.impala_beeswax.ImpalaBeeswaxResult object at 0x7353b50>.data
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org