[jira] [Commented] (IMPALA-5847) Some query options do not work as expected in .test files

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-5847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749012#comment-16749012 ] 

ASF subversion and git services commented on IMPALA-5847:
---------------------------------------------------------

Commit 6fa35478d5930cc3ec6d88c6b2e3f6bedd3d8ed9 in impala's branch refs/heads/master from Thomas Tauber-Marshall
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6fa3547 ]

IMPALA-5847: Fix incorrect use of SET in .test files

The '.test' files are used to run queries for tests. These files are
run with a vector of default query options. They also sometimes
include SET queries that modify query options. If SET is used on a
query option that is included in the vector, the default value from
the vector will override the value from the SET, leading to tests that
don't actually run with the query options they appear to.

This patch asserts that '.test' files don't use SET for values present
in the default vector. It also fixes various tests that already had
this incorrect behavior.

Testing:
- Passed a full exhaustive run.

Change-Id: I4e4c0f31bf4850642b624acdb1f6cb8837957990
Reviewed-on: http://gerrit.cloudera.org:8080/12220
Reviewed-by: Thomas Marshall <th...@cmu.edu>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Some query options do not work as expected in .test files
> ---------------------------------------------------------
>
>                 Key: IMPALA-5847
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5847
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Infrastructure
>            Reporter: Alexander Behm
>            Assignee: Thomas Tauber-Marshall
>            Priority: Minor
>
> We often use "set" in .test files to alter query options. Theoretically, a "set" command should change the session-level query options and in most cases a single .test file is executed from the same Impala session. However, for some options using "set" within a query section does not seem to work. For example, "num_nodes" does not work as expected as shown below.
> PyTest:
> {code}
> import pytest
> from tests.common.impala_test_suite import ImpalaTestSuite
> class TestStringQueries(ImpalaTestSuite):
>   @classmethod
>   def get_workload(cls):
>     return 'functional-query'
>   def test_set_bug(self, vector):
>     self.run_test_case('QueryTest/set_bug', vector)
> {code}
> Corresponding .test file:
> {code}
> ====
> ---- QUERY
> set num_nodes=1;
> select count(*) from functional.alltypes;
> select count(*) from functional.alltypes;
> select count(*) from functional.alltypes;
> ---- RESULTS
> 7300
> ---- TYPES
> BIGINT
> ====
> {code}
> After running the test above, I validated that the 3 queries were run from the same session, and that the queries run a distributed plan. The "num_nodes" option was definitely not picked up. I am not sure which query options are affected. In several .test files setting other query options does seem to work as expected.
> I suspect that the test framework might keep its own list of default query options which get submitted together with the query, so the session-level options are overridden on a per-request basis. For example, if I change the pytest to remove the "num_nodes" dictionary entry, then the test works as expected.
> PyTest workaround:
> {code}
> import pytest
> from tests.common.impala_test_suite import ImpalaTestSuite
> class TestStringQueries(ImpalaTestSuite):
>   @classmethod
>   def get_workload(cls):
>     return 'functional-query'
>   def test_set_bug(self, vector):
>     # Workaround SET bug
>     vector.get_value('exec_option').pop('num_nodes', None)
>     self.run_test_case('QueryTest/set_bug', vector)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org