You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Philip Zeyliger (Code Review)" <ge...@cloudera.org> on 2017/09/18 21:26:18 UTC

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Philip Zeyliger has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/8096

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................

Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

(Re-applies reverted commit 387bde0639ffd8ef580ccbf727152954e62bacbe.
The commit broke ASAN tests due to a race in how test infrastructure
re-uses connections. The fix for that is in an adjacent commit.)

When converting TQueryOptions to a map<string,string>, we now convert
unset options to the empty string. Within TQueryOptions we have optional
options (like mt_dop or compression_codec) with no default specified. In
this case, the user was seeing 0 for numeric types and the first enum
option for enumeration types (e.g., "NONE" in the compression case).
This was confusing as the implementation handles this "null" case
differently (e.g., using SNAPPY as the default codec in the case
reported in the JIRA).

When running "set" in impala-shell, the difference is as
follows:

    -        BUFFER_POOL_LIMIT: [0]
    +        BUFFER_POOL_LIMIT: []
    -        COMPRESSION_CODEC: [NONE]
    +        COMPRESSION_CODEC: []
    -        MT_DOP: [0]
    +        MT_DOP: []
    -        RESERVATION_REQUEST_TIMEOUT: [0]
    +        RESERVATION_REQUEST_TIMEOUT: []
    -        SEQ_COMPRESSION_MODE: [0]
    +        SEQ_COMPRESSION_MODE: []
    -        V_CPU_CORES: [0]
    +        V_CPU_CORES: []

Obviously, the empty string is a valid value for a string-typed option, where
it will be impossible to tell the difference between "unset" and "set to empty
string." Today, there are two string-typed options: debug_string defaults to ""
and request_pool has no default. An alternative would have been to use
a special token like "_unset" or to introduce a new field in the beeswax
Thrift ConfigVariable struct. I think the empty string approach
is clearest.

The other users of this state, which I believe are HiveServer2's OpenSession()
call and HiveServer2's response to a "SET" query are affected. They
benefit from the same fix, and a new test has been added to test_hs2.py.

I did a mild refactoring in the HS2 tests to write a helper method
for the very common pattern of excecuting a query.

Testing:
* Manual testing with impala-shell
* Modified impala-shell tests to check this explicitly for one case.
* Modified HS2 test to check this as well as the SET k=v statement,
  which looked otherwise untested.

Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/hs2/hs2_test_suite.py
M tests/hs2/test_hs2.py
M tests/shell/test_shell_commandline.py
7 files changed, 106 insertions(+), 60 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/8096/1
-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8096 )

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-Change-Number: 8096
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Oct 2017 01:11:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/8096 )

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................


Patch Set 3: Code-Review+2

Cool, thanks for clarifying.


-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-Change-Number: 8096
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Sep 2017 23:24:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8096 )

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1285/


-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-Change-Number: 8096
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Oct 2017 21:07:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8096 )

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................

Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

(Re-applies reverted commit 387bde0639ffd8ef580ccbf727152954e62bacbe.
The commit broke ASAN tests due to a race in how test infrastructure
re-uses connections. The fix for that is in an adjacent commit.)

When converting TQueryOptions to a map<string,string>, we now convert
unset options to the empty string. Within TQueryOptions we have optional
options (like mt_dop or compression_codec) with no default specified. In
this case, the user was seeing 0 for numeric types and the first enum
option for enumeration types (e.g., "NONE" in the compression case).
This was confusing as the implementation handles this "null" case
differently (e.g., using SNAPPY as the default codec in the case
reported in the JIRA).

When running "set" in impala-shell, the difference is as
follows:

    -        BUFFER_POOL_LIMIT: [0]
    +        BUFFER_POOL_LIMIT: []
    -        COMPRESSION_CODEC: [NONE]
    +        COMPRESSION_CODEC: []
    -        MT_DOP: [0]
    +        MT_DOP: []
    -        RESERVATION_REQUEST_TIMEOUT: [0]
    +        RESERVATION_REQUEST_TIMEOUT: []
    -        SEQ_COMPRESSION_MODE: [0]
    +        SEQ_COMPRESSION_MODE: []
    -        V_CPU_CORES: [0]
    +        V_CPU_CORES: []

Obviously, the empty string is a valid value for a string-typed option, where
it will be impossible to tell the difference between "unset" and "set to empty
string." Today, there are two string-typed options: debug_string defaults to ""
and request_pool has no default. An alternative would have been to use
a special token like "_unset" or to introduce a new field in the beeswax
Thrift ConfigVariable struct. I think the empty string approach
is clearest.

The other users of this state, which I believe are HiveServer2's OpenSession()
call and HiveServer2's response to a "SET" query are affected. They
benefit from the same fix, and a new test has been added to test_hs2.py.

I did a mild refactoring in the HS2 tests to write a helper method
for the very common pattern of excecuting a query.

Testing:
* Manual testing with impala-shell
* Modified impala-shell tests to check this explicitly for one case.
* Modified HS2 test to check this as well as the SET k=v statement,
  which looked otherwise untested.

Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Reviewed-on: http://gerrit.cloudera.org:8080/8096
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/hs2/hs2_test_suite.py
M tests/hs2/test_hs2.py
M tests/shell/test_shell_commandline.py
7 files changed, 106 insertions(+), 60 deletions(-)

Approvals:
  Dan Hecht: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-Change-Number: 8096
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/8096 )

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................


Patch Set 3:

But I'll wait to submit until change 8070 goes in, in case that one fails.


-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-Change-Number: 8096
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Sep 2017 23:25:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/8096 )

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................


Patch Set 4: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-Change-Number: 8096
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Oct 2017 21:07:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8096 )

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................


Patch Set 3:

> Patch Set 3:
> 
> Is this the same patch as before (I.e. can we just carry forward that +2), or did something in this diff change since the +2?

This is a computer-generated revert-of-a-revert with a slight change to the commit message. 

I double checked with git patch-id; they're the same diff. 

$git log --oneline --grep IMPALA-5589 | grep -v Revert
598b72c Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
387bde0 IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

$git show 387bde0 | git-patch-id --stable; git show 598b72c | git-patch-id --stable
47b02290d8b30ffa520ea19dac8353075c5a5bc7 387bde0639ffd8ef580ccbf727152954e62bacbe
47b02290d8b30ffa520ea19dac8353075c5a5bc7 598b72cf43a326dd3c8a9a35994f8007db665b93


-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-Change-Number: 8096
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Sep 2017 22:56:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8096 )

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................


Patch Set 3:

> Patch Set 3:
> 
> > Patch Set 3:
> > 
> > Is this the same patch as before (I.e. can we just carry forward that +2), or did something in this diff change since the +2?
> 
> This is a computer-generated revert-of-a-revert with a slight change to the commit message. 
> 
> I double checked with git patch-id; they're the same diff. 
> 
> $git log --oneline --grep IMPALA-5589 | grep -v Revert
> 598b72c Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
> 387bde0 IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
> 
> $git show 387bde0 | git-patch-id --stable; git show 598b72c | git-patch-id --stable
> 47b02290d8b30ffa520ea19dac8353075c5a5bc7 387bde0639ffd8ef580ccbf727152954e62bacbe
> 47b02290d8b30ffa520ea19dac8353075c5a5bc7 598b72cf43a326dd3c8a9a35994f8007db665b93

And, yes, I think we should carry forward the +2.


-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-Change-Number: 8096
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Sep 2017 22:57:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change.

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................


Patch Set 1:

This is a re-run of http://gerrit.cloudera.org:8080/7886. The interesting thing is https://gerrit.cloudera.org/#/c/8070.

(I'm happy to collapse the two changes if that's preferred.)

-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options

Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/8096 )

Change subject: Re-apply: IMPALA-5589: change "set" in impala-shell to show empty string for unset query options
......................................................................


Patch Set 3:

Is this the same patch as before (I.e. can we just carry forward that +2), or did something in this diff change since the +2?


-- 
To view, visit http://gerrit.cloudera.org:8080/8096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29f5d8ab874cb1338077f16019a9537766cac0c4
Gerrit-Change-Number: 8096
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Sep 2017 22:08:17 +0000
Gerrit-HasComments: No