You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Jason Fehr (Code Review)" <ge...@cloudera.org> on 2023/01/03 20:34:56 UTC

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Jason Fehr has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19388


Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................

IMPALA-11375 Impala shell outputs details of each RPC

When the Impala shell is using the hs2 protocol, it makes multiple RPCs
to the Impala daemon.  These calls pass Thrift objects back and forth.
This change adds the '--show_rpc' which outputs the details of the RPCs
to stdout and the '--rpc_file' flag which outputs the RPC details to the
specified file path.

RPC details include:
- operation name
- request attempt count
- Impala session/query ids (if applicable)
- call duration
- call status (success/failure)
- request Thrift objects
- response Thrift objects

Certain information is not included in the RPC details:
- Thrift object attributes named 'secret' or 'password'
  are redacted.
- Thrift objects with a type of TRowSet or TGetRuntimeProfileResp
  are not include as the information contained within them is
  already available in the standars output from the Impala shell.

Testing:
- Added new tests in the end-to-end test suite.

Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
---
M bin/impala-shell.sh
M shell/impala_client.py
M shell/impala_shell.py
M shell/impala_shell_config_defaults.py
M shell/make_shell_tarball.sh
M shell/option_parser.py
M shell/packaging/make_python_package.sh
A shell/thrift_printer.py
M tests/shell/test_shell_commandline.py
9 files changed, 531 insertions(+), 152 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/5
-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 5
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 5:

(108 comments)

http://gerrit.cloudera.org:8080/#/c/19388/5/bin/impala-shell.sh
File bin/impala-shell.sh:

http://gerrit.cloudera.org:8080/#/c/19388/5/bin/impala-shell.sh@57
PS5, Line 57:   echo "impala python shell waiting for remote debugging connection on port ${IMPALA_SHELL_DEBUG_PORT}"
line too long (103 > 90)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py
File shell/impala_client.py:

http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@684
PS5, Line 684: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@810
PS5, Line 810: o
flake8: E501 line too long (99 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@820
PS5, Line 820: y
flake8: E501 line too long (106 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@840
PS5, Line 840: =
flake8: E711 comparison to None should be 'if cond is None:'


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@884
PS5, Line 884: e
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@934
PS5, Line 934: n
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@975
PS5, Line 975: r
flake8: E501 line too long (107 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@987
PS5, Line 987: )
flake8: E501 line too long (113 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1002
PS5, Line 1002: t
flake8: E501 line too long (99 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1021
PS5, Line 1021: T
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1067
PS5, Line 1067: e
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1181
PS5, Line 1181: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1185
PS5, Line 1185: -
flake8: E501 line too long (118 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1187
PS5, Line 1187: ,
flake8: E231 missing whitespace after ','


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1193
PS5, Line 1193: -
flake8: E501 line too long (118 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1210
PS5, Line 1210: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1215
PS5, Line 1215: -
flake8: E501 line too long (118 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1217
PS5, Line 1217: ,
flake8: E231 missing whitespace after ','


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1220
PS5, Line 1220: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1223
PS5, Line 1223: -
flake8: E501 line too long (118 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/impala_client.py@1545
PS5, Line 1545: 
flake8: W292 no newline at end of file


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/thrift_printer.py
File shell/thrift_printer.py:

http://gerrit.cloudera.org:8080/#/c/19388/5/shell/thrift_printer.py@23
PS5, Line 23: class ThriftPrettyPrinter(object):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/thrift_printer.py@38
PS5, Line 38: ,
flake8: E231 missing whitespace after ','


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/thrift_printer.py@40
PS5, Line 40: ,
flake8: E231 missing whitespace after ','


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/thrift_printer.py@41
PS5, Line 41: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/thrift_printer.py@128
PS5, Line 128: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/5/shell/thrift_printer.py@130
PS5, Line 130: =
flake8: E711 comparison to None should be 'if cond is None:'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:

http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1424
PS5, Line 1424: e
flake8: E501 line too long (111 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1425
PS5, Line 1425: \
flake8: E501 line too long (130 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1429
PS5, Line 1429: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1429
PS5, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1429
PS5, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1429
PS5, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1429
PS5, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1429
PS5, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1429
PS5, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1429
PS5, Line 1429: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1429
PS5, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1429
PS5, Line 1429: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1432
PS5, Line 1432: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1433
PS5, Line 1433: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1434
PS5, Line 1434: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1437
PS5, Line 1437: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1437
PS5, Line 1437: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1446
PS5, Line 1446: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1446
PS5, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1446
PS5, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1446
PS5, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1446
PS5, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1446
PS5, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1446
PS5, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1446
PS5, Line 1446: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1446
PS5, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1446
PS5, Line 1446: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1449
PS5, Line 1449: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1449
PS5, Line 1449: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1450
PS5, Line 1450: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1453
PS5, Line 1453: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1453
PS5, Line 1453: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1456
PS5, Line 1456: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1456
PS5, Line 1456: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1457
PS5, Line 1457: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1457
PS5, Line 1457: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1458
PS5, Line 1458: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1458
PS5, Line 1458: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1459
PS5, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1459
PS5, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1459
PS5, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1459
PS5, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1459
PS5, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1459
PS5, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1459
PS5, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1460
PS5, Line 1460: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1460
PS5, Line 1460: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1461
PS5, Line 1461: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1461
PS5, Line 1461: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1462
PS5, Line 1462: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1462
PS5, Line 1462: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1463
PS5, Line 1463: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1463
PS5, Line 1463: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1464
PS5, Line 1464: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1464
PS5, Line 1464: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1470
PS5, Line 1470: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1470
PS5, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1470
PS5, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1470
PS5, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1470
PS5, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1470
PS5, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1470
PS5, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1470
PS5, Line 1470: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1470
PS5, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1470
PS5, Line 1470: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1473
PS5, Line 1473: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1474
PS5, Line 1474: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1479
PS5, Line 1479: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1479
PS5, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1479
PS5, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1479
PS5, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1479
PS5, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1479
PS5, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1479
PS5, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1479
PS5, Line 1479: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1479
PS5, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1479
PS5, Line 1479: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1482
PS5, Line 1482: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1482
PS5, Line 1482: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/5/tests/shell/test_shell_commandline.py@1483
PS5, Line 1483: \
flake8: W605 invalid escape sequence '\*'



-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 5
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jan 2023 20:35:52 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 16:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8963/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 16
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Jan 2023 18:19:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 16: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 16
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Jan 2023 18:19:53 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375: Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375: Impala shell outputs details of each RPC
......................................................................


Patch Set 13: Code-Review+1

carry forward +1 for very minor code modification


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 13
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Jan 2023 23:13:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 11: Code-Review+1

(3 comments)

LGTM

http://gerrit.cloudera.org:8080/#/c/19388/11//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/19388/11//COMMIT_MSG@2
PS11, Line 2: Author:     jasonmfehr <ja...@gmail.com>
Nit: Author still shows gmail address


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py
File shell/impala_client.py:

http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@776
PS9, Line 776:       key_value_pairs = set_query_options.items()
> That's actually the formatting used elsewhere, thus I did not change it.
ok


http://gerrit.cloudera.org:8080/#/c/19388/11/shell/option_parser.py
File shell/option_parser.py:

http://gerrit.cloudera.org:8080/#/c/19388/11/shell/option_parser.py@216
PS11, Line 216:                     "file.  Ignored if protocol is beeswax.")
Nit: delete double space



-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 11
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 20:16:05 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 11:

(2 comments)

addressed comments

http://gerrit.cloudera.org:8080/#/c/19388/11//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/19388/11//COMMIT_MSG@2
PS11, Line 2: Author:     jasonmfehr <ja...@gmail.com>
> Nit: Author still shows gmail address
I was going to fix this on future gerrit changes only, but since there is another code change needed on this PR I am going to amend the commit and fix the email.


http://gerrit.cloudera.org:8080/#/c/19388/11/shell/option_parser.py
File shell/option_parser.py:

http://gerrit.cloudera.org:8080/#/c/19388/11/shell/option_parser.py@216
PS11, Line 216:                     "file.  Ignored if protocol is beeswax.")
> Nit: delete double space
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 11
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 21:15:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 16: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 16
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Jan 2023 23:31:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12109/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 11
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 18:20:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 12: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/19388/12//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/19388/12//COMMIT_MSG@7
PS12, Line 7: IMPALA-11375 Impala shell outputs details of each RPC
Per coding standards the title should have a colon after the JIRA Id:
IMPALA-11375: Impala shell outputs details of each RPC



-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 12
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Jan 2023 22:23:14 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................

IMPALA-11375 Impala shell outputs details of each RPC

When the Impala shell is using the hs2 protocol, it makes multiple RPCs
to the Impala daemon.  These calls pass Thrift objects back and forth.
This change adds the '--show_rpc' which outputs the details of the RPCs
to stdout and the '--rpc_file' flag which outputs the RPC details to the
specified file path.

RPC details include:
- operation name
- request attempt count
- Impala session/query ids (if applicable)
- call duration
- call status (success/failure)
- request Thrift objects
- response Thrift objects

Certain information is not included in the RPC details:
- Thrift object attributes named 'secret' or 'password'
  are redacted.
- Thrift objects with a type of TRowSet or TGetRuntimeProfileResp
  are not include as the information contained within them is
  already available in the standard output from the Impala shell.

Testing:
- Added new tests in the end-to-end test suite.

Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
---
M bin/impala-shell.sh
M shell/impala_client.py
M shell/impala_shell.py
M shell/impala_shell_config_defaults.py
M shell/make_shell_tarball.sh
M shell/option_parser.py
M shell/packaging/make_python_package.sh
A shell/thrift_printer.py
M tests/shell/test_shell_commandline.py
9 files changed, 565 insertions(+), 147 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/15
-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 15
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375: Impala shell outputs details of each RPC

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375: Impala shell outputs details of each RPC
......................................................................


Patch Set 13: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 13
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Jan 2023 23:14:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 9:

(22 comments)

Generally looks good...

http://gerrit.cloudera.org:8080/#/c/19388/9//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/19388/9//COMMIT_MSG@2
PS9, Line 2: Author:     jasonmfehr <ja...@gmail.com>
If you want you can use a cloudera address


http://gerrit.cloudera.org:8080/#/c/19388/9//COMMIT_MSG@29
PS9, Line 29:   already available in the standars output from the Impala shell.
Nit: "standard"


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py
File shell/impala_client.py:

http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@701
PS9, Line 701:     resp = self._do_hs2_rpc(
Nit: I think this style looks neater
    req = TOpenSessionReq(TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V6,
                          username=self.user)
    resp = self._do_hs2_rpc(OpenSession, req, retry_on_error=True)


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@776
PS9, Line 776:     # PingImpalaHS2Service rpc is idempotent and so safe to retry.
Nit: I expected a blank line before this


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@829
PS9, Line 829:     self._clear_current_query_id()
This method seems unusual in that it doesn't use try...finally for the current_query_id, why is that?


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@837
PS9, Line 837:     self._current_query_id = self.get_query_id_str(query_handle)
We always calculate the _current_query_id even when show_rpc or rpc_file are not set. I think we should aim to minimize the impact on people not using the new feature. Could we just save the query_handle? We could call _convert_id_to_str only when we need the string in print_start_to_file().


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@1216
PS9, Line 1216:         f = open(self.rpc_file, "a")
It looks like we always open the file for append, so if I specify an existing file with --rpc_file I think it will not be overwritten? That seems a little unintuitive.


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@1218
PS9, Line 1218:         f.close()
Better maybe:
  with open(self.rpc_file, "a") as f:
    print_start_to_file(f)


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@1246
PS9, Line 1246:         f = open(self.rpc_file, "a")
Use "with" as noted above?


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py
File shell/option_parser.py:

http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py@209
PS9, Line 209:   parser.add_option("--show_rpc", dest="show_rpc",
Reading this I thought that I always needed to set show_rpc, and that I set rpc_file if I want output in a file.
But IIUIC if I set both then output goes both to file and stdout.
Ideally this should be clearer
One way is to have --rpc_stdout and --rpc_file the I can see they are doing the same thing.


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py@211
PS9, Line 211:                     help="Always show hs2 rpc details.  Ignored if protocol is beeswax")
Nit: use 1 space?
Why is is "Always"? Maybe we don't need that word.


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py@213
PS9, Line 213:                     help="If set, hs2 rpc call details are writte to the given file.  "
Nit: "written"


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py
File shell/thrift_printer.py:

http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@27
PS9, Line 27:   to stdout, stderrr, or any other file handles."""
Nit: "stderr"


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@30
PS9, Line 30:   #   redacted_fields - list of names of object attributes who's
Nit: "whose"


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@33
PS9, Line 33:   #                     base indentation where all other indetations
Nit: "indentations"


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@36
PS9, Line 36:   #                     will not be be printed out, useful to ignore
Nit: "will not be"


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@56
PS9, Line 56:     be anything that has a write(string) method on it.
Nit: just "has a write(string) method" may be clearer?


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@64
PS9, Line 64:     #   thrfit_obj  - the object to print out, its attributes will
Nit: "thrift_obj"


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@69
PS9, Line 69:     #                 write(str) method on it
Nit: just "has a write(string) method" may be clearer?


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@96
PS9, Line 96:       # TODO -- print out attributes in alphabetical order
Looks like this is done already?


http://gerrit.cloudera.org:8080/#/c/19388/9/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:

http://gerrit.cloudera.org:8080/#/c/19388/9/tests/shell/test_shell_commandline.py@1412
PS9, Line 1412:   def test_output_rpc_to_screen(self, vector, populated_table, tmp_file):
Add a brief description of what the test does.


http://gerrit.cloudera.org:8080/#/c/19388/9/tests/shell/test_shell_commandline.py@1422
PS9, Line 1422:     def check_multiline(assert_str, regex_lines):
Add one line description of function



-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 23:09:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................

IMPALA-11375 Impala shell outputs details of each RPC

When the Impala shell is using the hs2 protocol, it makes multiple RPCs
to the Impala daemon.  These calls pass Thrift objects back and forth.
This change adds the '--show_rpc' which outputs the details of the RPCs
to stdout and the '--rpc_file' flag which outputs the RPC details to the
specified file path.

RPC details include:
- operation name
- request attempt count
- Impala session/query ids (if applicable)
- call duration
- call status (success/failure)
- request Thrift objects
- response Thrift objects

Certain information is not included in the RPC details:
- Thrift object attributes named 'secret' or 'password'
  are redacted.
- Thrift objects with a type of TRowSet or TGetRuntimeProfileResp
  are not include as the information contained within them is
  already available in the standard output from the Impala shell.

Testing:
- Added new tests in the end-to-end test suite.

Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
---
M bin/impala-shell.sh
M shell/impala_client.py
M shell/impala_shell.py
M shell/impala_shell_config_defaults.py
M shell/make_shell_tarball.sh
M shell/option_parser.py
M shell/packaging/make_python_package.sh
A shell/thrift_printer.py
M tests/shell/test_shell_commandline.py
9 files changed, 556 insertions(+), 147 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/12
-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 12
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12110/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 12
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 21:35:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375: Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Hello Andrew Sherman, Abhishek Rawat, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19388

to look at the new patch set (#13).

Change subject: IMPALA-11375: Impala shell outputs details of each RPC
......................................................................

IMPALA-11375: Impala shell outputs details of each RPC

When the Impala shell is using the hs2 protocol, it makes multiple RPCs
to the Impala daemon.  These calls pass Thrift objects back and forth.
This change adds the '--show_rpc' which outputs the details of the RPCs
to stdout and the '--rpc_file' flag which outputs the RPC details to the
specified file path.

RPC details include:
- operation name
- request attempt count
- Impala session/query ids (if applicable)
- call duration
- call status (success/failure)
- request Thrift objects
- response Thrift objects

Certain information is not included in the RPC details:
- Thrift object attributes named 'secret' or 'password'
  are redacted.
- Thrift objects with a type of TRowSet or TGetRuntimeProfileResp
  are not include as the information contained within them is
  already available in the standard output from the Impala shell.

Testing:
- Added new tests in the end-to-end test suite.

Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
---
M bin/impala-shell.sh
M shell/impala_client.py
M shell/impala_shell.py
M shell/impala_shell_config_defaults.py
M shell/make_shell_tarball.sh
M shell/option_parser.py
M shell/packaging/make_python_package.sh
A shell/thrift_printer.py
M tests/shell/test_shell_commandline.py
9 files changed, 556 insertions(+), 147 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/13
-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 13
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12089/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jan 2023 23:55:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 12: Code-Review+1

carry forward +1 for very minor code modification


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 12
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 22:35:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 15: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 15
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Jan 2023 18:19:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 9:

(22 comments)

comments addressed

http://gerrit.cloudera.org:8080/#/c/19388/9//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/19388/9//COMMIT_MSG@2
PS9, Line 2: Author:     jasonmfehr <ja...@gmail.com>
> If you want you can use a cloudera address
Next change will use my cloudera email address.


http://gerrit.cloudera.org:8080/#/c/19388/9//COMMIT_MSG@29
PS9, Line 29:   already available in the standars output from the Impala shell.
> Nit: "standard"
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py
File shell/impala_client.py:

http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@701
PS9, Line 701:     resp = self._do_hs2_rpc(
> Nit: I think this style looks neater
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@776
PS9, Line 776:     # PingImpalaHS2Service rpc is idempotent and so safe to retry.
> Nit: I expected a blank line before this
That's actually the formatting used elsewhere, thus I did not change it.


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@829
PS9, Line 829:     self._clear_current_query_id()
> This method seems unusual in that it doesn't use try...finally for the curr
It was a miss on my part because this method is different.  Adding the try...finally


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@837
PS9, Line 837:     self._current_query_id = self.get_query_id_str(query_handle)
> We always calculate the _current_query_id even when show_rpc or rpc_file ar
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@1216
PS9, Line 1216:         f = open(self.rpc_file, "a")
> It looks like we always open the file for append, so if I specify an existi
The file is opened in append mode.  That way, it behaves more like a log file.  It's the safer option because existing data does not get deleted.  If a user accidentally specifies the wrong output file, their data won't be erased.

I added a sentence to the help text of the '--rpc_file' command line option explaining that it appends to existing files.


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@1218
PS9, Line 1218:         f.close()
> Better maybe:
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/impala_client.py@1246
PS9, Line 1246:         f = open(self.rpc_file, "a")
> Use "with" as noted above?
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py
File shell/option_parser.py:

http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py@209
PS9, Line 209:   parser.add_option("--show_rpc", dest="show_rpc",
> Reading this I thought that I always needed to set show_rpc, and that I set
I like this idea.  Changing '--show_rpc' to '--rpc_stdout' and making changes to the help text to make it clearer.


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py@211
PS9, Line 211:                     help="Always show hs2 rpc details.  Ignored if protocol is beeswax")
> Nit: use 1 space?
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/option_parser.py@213
PS9, Line 213:                     help="If set, hs2 rpc call details are writte to the given file.  "
> Nit: "written"
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py
File shell/thrift_printer.py:

http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@27
PS9, Line 27:   to stdout, stderrr, or any other file handles."""
> Nit: "stderr"
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@30
PS9, Line 30:   #   redacted_fields - list of names of object attributes who's
> Nit: "whose"
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@33
PS9, Line 33:   #                     base indentation where all other indetations
> Nit: "indentations"
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@36
PS9, Line 36:   #                     will not be be printed out, useful to ignore
> Nit: "will not be"
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@56
PS9, Line 56:     be anything that has a write(string) method on it.
> Nit: just "has a write(string) method" may be clearer?
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@64
PS9, Line 64:     #   thrfit_obj  - the object to print out, its attributes will
> Nit: "thrift_obj"
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@69
PS9, Line 69:     #                 write(str) method on it
> Nit: just "has a write(string) method" may be clearer?
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/shell/thrift_printer.py@96
PS9, Line 96:       # TODO -- print out attributes in alphabetical order
> Looks like this is done already?
yes, removed comment


http://gerrit.cloudera.org:8080/#/c/19388/9/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:

http://gerrit.cloudera.org:8080/#/c/19388/9/tests/shell/test_shell_commandline.py@1412
PS9, Line 1412:   def test_output_rpc_to_screen(self, vector, populated_table, tmp_file):
> Add a brief description of what the test does.
Done


http://gerrit.cloudera.org:8080/#/c/19388/9/tests/shell/test_shell_commandline.py@1422
PS9, Line 1422:     def check_multiline(assert_str, regex_lines):
> Add one line description of function
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 17:52:52 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 15:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12142/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 15
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Jan 2023 21:56:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375: Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375: Impala shell outputs details of each RPC
......................................................................


Patch Set 14:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8956/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 14
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Jan 2023 23:16:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12087/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 7
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jan 2023 22:33:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12085/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 5
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jan 2023 20:56:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375: Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375: Impala shell outputs details of each RPC
......................................................................


Patch Set 14: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 14
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Jan 2023 23:16:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375: Impala shell outputs details of each RPC

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has removed a vote on this change.

Change subject: IMPALA-11375: Impala shell outputs details of each RPC
......................................................................


Removed Code-Review+2 by Impala Public Jenkins <im...@cloudera.com>
-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 14
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................

IMPALA-11375 Impala shell outputs details of each RPC

When the Impala shell is using the hs2 protocol, it makes multiple RPCs
to the Impala daemon.  These calls pass Thrift objects back and forth.
This change adds the '--show_rpc' which outputs the details of the RPCs
to stdout and the '--rpc_file' flag which outputs the RPC details to the
specified file path.

RPC details include:
- operation name
- request attempt count
- Impala session/query ids (if applicable)
- call duration
- call status (success/failure)
- request Thrift objects
- response Thrift objects

Certain information is not included in the RPC details:
- Thrift object attributes named 'secret' or 'password'
  are redacted.
- Thrift objects with a type of TRowSet or TGetRuntimeProfileResp
  are not include as the information contained within them is
  already available in the standars output from the Impala shell.

Testing:
- Added new tests in the end-to-end test suite.

Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
---
M bin/impala-shell.sh
M shell/impala_client.py
M shell/impala_shell.py
M shell/impala_shell_config_defaults.py
M shell/make_shell_tarball.sh
M shell/option_parser.py
M shell/packaging/make_python_package.sh
A shell/thrift_printer.py
M tests/shell/test_shell_commandline.py
9 files changed, 530 insertions(+), 151 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/7
-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 7
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 8:

(80 comments)

http://gerrit.cloudera.org:8080/#/c/19388/8/shell/impala_client.py
File shell/impala_client.py:

http://gerrit.cloudera.org:8080/#/c/19388/8/shell/impala_client.py@1013
PS8, Line 1013: r
flake8: E131 continuation line unaligned for hanging indent


http://gerrit.cloudera.org:8080/#/c/19388/8/shell/impala_client.py@1206
PS8, Line 1206: ,
flake8: E231 missing whitespace after ','


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:

http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1433
PS8, Line 1433: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1433
PS8, Line 1433: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1433
PS8, Line 1433: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1433
PS8, Line 1433: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1433
PS8, Line 1433: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1433
PS8, Line 1433: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1433
PS8, Line 1433: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1433
PS8, Line 1433: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1433
PS8, Line 1433: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1433
PS8, Line 1433: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1436
PS8, Line 1436: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1437
PS8, Line 1437: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1438
PS8, Line 1438: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1441
PS8, Line 1441: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1441
PS8, Line 1441: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1450
PS8, Line 1450: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1450
PS8, Line 1450: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1450
PS8, Line 1450: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1450
PS8, Line 1450: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1450
PS8, Line 1450: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1450
PS8, Line 1450: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1450
PS8, Line 1450: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1450
PS8, Line 1450: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1450
PS8, Line 1450: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1450
PS8, Line 1450: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1453
PS8, Line 1453: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1453
PS8, Line 1453: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1454
PS8, Line 1454: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1457
PS8, Line 1457: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1457
PS8, Line 1457: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1460
PS8, Line 1460: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1460
PS8, Line 1460: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1461
PS8, Line 1461: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1461
PS8, Line 1461: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1462
PS8, Line 1462: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1462
PS8, Line 1462: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1463
PS8, Line 1463: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1463
PS8, Line 1463: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1463
PS8, Line 1463: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1463
PS8, Line 1463: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1463
PS8, Line 1463: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1463
PS8, Line 1463: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1463
PS8, Line 1463: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1464
PS8, Line 1464: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1464
PS8, Line 1464: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1465
PS8, Line 1465: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1465
PS8, Line 1465: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1466
PS8, Line 1466: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1466
PS8, Line 1466: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1467
PS8, Line 1467: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1467
PS8, Line 1467: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1468
PS8, Line 1468: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1468
PS8, Line 1468: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1474
PS8, Line 1474: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1474
PS8, Line 1474: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1474
PS8, Line 1474: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1474
PS8, Line 1474: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1474
PS8, Line 1474: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1474
PS8, Line 1474: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1474
PS8, Line 1474: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1474
PS8, Line 1474: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1474
PS8, Line 1474: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1474
PS8, Line 1474: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1477
PS8, Line 1477: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1478
PS8, Line 1478: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1483
PS8, Line 1483: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1483
PS8, Line 1483: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1483
PS8, Line 1483: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1483
PS8, Line 1483: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1483
PS8, Line 1483: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1483
PS8, Line 1483: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1483
PS8, Line 1483: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1483
PS8, Line 1483: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1483
PS8, Line 1483: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1483
PS8, Line 1483: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1486
PS8, Line 1486: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1486
PS8, Line 1486: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/8/tests/shell/test_shell_commandline.py@1487
PS8, Line 1487: \
flake8: W605 invalid escape sequence '\*'



-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jan 2023 23:34:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 7:

(108 comments)

http://gerrit.cloudera.org:8080/#/c/19388/7/bin/impala-shell.sh
File bin/impala-shell.sh:

http://gerrit.cloudera.org:8080/#/c/19388/7/bin/impala-shell.sh@57
PS7, Line 57:   echo "impala python shell waiting for remote debugging connection on port ${IMPALA_SHELL_DEBUG_PORT}"
line too long (103 > 90)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py
File shell/impala_client.py:

http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@684
PS7, Line 684: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@810
PS7, Line 810: o
flake8: E501 line too long (99 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@820
PS7, Line 820: y
flake8: E501 line too long (106 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@840
PS7, Line 840: =
flake8: E711 comparison to None should be 'if cond is None:'


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@884
PS7, Line 884: e
flake8: E501 line too long (91 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@934
PS7, Line 934: n
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@975
PS7, Line 975: r
flake8: E501 line too long (107 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@987
PS7, Line 987: )
flake8: E501 line too long (113 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1002
PS7, Line 1002: t
flake8: E501 line too long (99 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1021
PS7, Line 1021: T
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1067
PS7, Line 1067: e
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1181
PS7, Line 1181: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1185
PS7, Line 1185: -
flake8: E501 line too long (118 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1187
PS7, Line 1187: ,
flake8: E231 missing whitespace after ','


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1193
PS7, Line 1193: -
flake8: E501 line too long (118 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1210
PS7, Line 1210: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1215
PS7, Line 1215: -
flake8: E501 line too long (118 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1217
PS7, Line 1217: ,
flake8: E231 missing whitespace after ','


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1220
PS7, Line 1220: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1223
PS7, Line 1223: -
flake8: E501 line too long (118 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/impala_client.py@1545
PS7, Line 1545: 
flake8: W292 no newline at end of file


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/thrift_printer.py
File shell/thrift_printer.py:

http://gerrit.cloudera.org:8080/#/c/19388/7/shell/thrift_printer.py@23
PS7, Line 23: class ThriftPrettyPrinter(object):
flake8: E302 expected 2 blank lines, found 1


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/thrift_printer.py@38
PS7, Line 38: ,
flake8: E231 missing whitespace after ','


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/thrift_printer.py@40
PS7, Line 40: ,
flake8: E231 missing whitespace after ','


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/thrift_printer.py@41
PS7, Line 41: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/thrift_printer.py@128
PS7, Line 128: !
flake8: E711 comparison to None should be 'if cond is not None:'


http://gerrit.cloudera.org:8080/#/c/19388/7/shell/thrift_printer.py@130
PS7, Line 130: =
flake8: E711 comparison to None should be 'if cond is None:'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:

http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1424
PS7, Line 1424: e
flake8: E501 line too long (111 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1425
PS7, Line 1425: \
flake8: E501 line too long (130 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1429
PS7, Line 1429: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1429
PS7, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1429
PS7, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1429
PS7, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1429
PS7, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1429
PS7, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1429
PS7, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1429
PS7, Line 1429: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1429
PS7, Line 1429: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1429
PS7, Line 1429: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1432
PS7, Line 1432: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1433
PS7, Line 1433: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1434
PS7, Line 1434: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1437
PS7, Line 1437: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1437
PS7, Line 1437: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1446
PS7, Line 1446: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1446
PS7, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1446
PS7, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1446
PS7, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1446
PS7, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1446
PS7, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1446
PS7, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1446
PS7, Line 1446: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1446
PS7, Line 1446: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1446
PS7, Line 1446: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1449
PS7, Line 1449: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1449
PS7, Line 1449: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1450
PS7, Line 1450: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1453
PS7, Line 1453: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1453
PS7, Line 1453: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1456
PS7, Line 1456: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1456
PS7, Line 1456: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1457
PS7, Line 1457: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1457
PS7, Line 1457: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1458
PS7, Line 1458: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1458
PS7, Line 1458: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1459
PS7, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1459
PS7, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1459
PS7, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1459
PS7, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1459
PS7, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1459
PS7, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1459
PS7, Line 1459: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1460
PS7, Line 1460: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1460
PS7, Line 1460: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1461
PS7, Line 1461: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1461
PS7, Line 1461: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1462
PS7, Line 1462: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1462
PS7, Line 1462: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1463
PS7, Line 1463: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1463
PS7, Line 1463: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1464
PS7, Line 1464: \
flake8: W605 invalid escape sequence '\<'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1464
PS7, Line 1464: \
flake8: W605 invalid escape sequence '\>'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1470
PS7, Line 1470: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1470
PS7, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1470
PS7, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1470
PS7, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1470
PS7, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1470
PS7, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1470
PS7, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1470
PS7, Line 1470: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1470
PS7, Line 1470: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1470
PS7, Line 1470: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1473
PS7, Line 1473: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1474
PS7, Line 1474: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1479
PS7, Line 1479: \
flake8: W605 invalid escape sequence '\['


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1479
PS7, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1479
PS7, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1479
PS7, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1479
PS7, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1479
PS7, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1479
PS7, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1479
PS7, Line 1479: \
flake8: W605 invalid escape sequence '\.'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1479
PS7, Line 1479: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1479
PS7, Line 1479: \
flake8: W605 invalid escape sequence '\]'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1482
PS7, Line 1482: \
flake8: W605 invalid escape sequence '\*'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1482
PS7, Line 1482: \
flake8: W605 invalid escape sequence '\d'


http://gerrit.cloudera.org:8080/#/c/19388/7/tests/shell/test_shell_commandline.py@1483
PS7, Line 1483: \
flake8: W605 invalid escape sequence '\*'



-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 7
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jan 2023 22:13:03 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................

IMPALA-11375 Impala shell outputs details of each RPC

When the Impala shell is using the hs2 protocol, it makes multiple RPCs
to the Impala daemon.  These calls pass Thrift objects back and forth.
This change adds the '--show_rpc' which outputs the details of the RPCs
to stdout and the '--rpc_file' flag which outputs the RPC details to the
specified file path.

RPC details include:
- operation name
- request attempt count
- Impala session/query ids (if applicable)
- call duration
- call status (success/failure)
- request Thrift objects
- response Thrift objects

Certain information is not included in the RPC details:
- Thrift object attributes named 'secret' or 'password'
  are redacted.
- Thrift objects with a type of TRowSet or TGetRuntimeProfileResp
  are not include as the information contained within them is
  already available in the standars output from the Impala shell.

Testing:
- Added new tests in the end-to-end test suite.

Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
---
M bin/impala-shell.sh
M shell/impala_client.py
M shell/impala_shell.py
M shell/impala_shell_config_defaults.py
M shell/make_shell_tarball.sh
M shell/option_parser.py
M shell/packaging/make_python_package.sh
A shell/thrift_printer.py
M tests/shell/test_shell_commandline.py
9 files changed, 562 insertions(+), 150 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/9
-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12108/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 10
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 18:11:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................

IMPALA-11375 Impala shell outputs details of each RPC

When the Impala shell is using the hs2 protocol, it makes multiple RPCs
to the Impala daemon.  These calls pass Thrift objects back and forth.
This change adds the '--show_rpc' which outputs the details of the RPCs
to stdout and the '--rpc_file' flag which outputs the RPC details to the
specified file path.

RPC details include:
- operation name
- request attempt count
- Impala session/query ids (if applicable)
- call duration
- call status (success/failure)
- request Thrift objects
- response Thrift objects

Certain information is not included in the RPC details:
- Thrift object attributes named 'secret' or 'password'
  are redacted.
- Thrift objects with a type of TRowSet or TGetRuntimeProfileResp
  are not include as the information contained within them is
  already available in the standard output from the Impala shell.

Testing:
- Added new tests in the end-to-end test suite.

Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
---
M bin/impala-shell.sh
M shell/impala_client.py
M shell/impala_shell.py
M shell/impala_shell_config_defaults.py
M shell/make_shell_tarball.sh
M shell/option_parser.py
M shell/packaging/make_python_package.sh
A shell/thrift_printer.py
M tests/shell/test_shell_commandline.py
9 files changed, 556 insertions(+), 147 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/10
-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 10
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 10:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/19388/10/shell/impala_client.py
File shell/impala_client.py:

http://gerrit.cloudera.org:8080/#/c/19388/10/shell/impala_client.py@1258
PS10, Line 1258:     
flake8: W293 blank line contains whitespace


http://gerrit.cloudera.org:8080/#/c/19388/10/shell/impala_client.py@1258
PS10, Line 1258:     
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 10
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jan 2023 17:52:55 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................

IMPALA-11375 Impala shell outputs details of each RPC

When the Impala shell is using the hs2 protocol, it makes multiple RPCs
to the Impala daemon.  These calls pass Thrift objects back and forth.
This change adds the '--show_rpc' which outputs the details of the RPCs
to stdout and the '--rpc_file' flag which outputs the RPC details to the
specified file path.

RPC details include:
- operation name
- request attempt count
- Impala session/query ids (if applicable)
- call duration
- call status (success/failure)
- request Thrift objects
- response Thrift objects

Certain information is not included in the RPC details:
- Thrift object attributes named 'secret' or 'password'
  are redacted.
- Thrift objects with a type of TRowSet or TGetRuntimeProfileResp
  are not include as the information contained within them is
  already available in the standard output from the Impala shell.

Testing:
- Added new tests in the end-to-end test suite.

Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Reviewed-on: http://gerrit.cloudera.org:8080/19388
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M bin/impala-shell.sh
M shell/impala_client.py
M shell/impala_shell.py
M shell/impala_shell_config_defaults.py
M shell/make_shell_tarball.sh
M shell/option_parser.py
M shell/packaging/make_python_package.sh
A shell/thrift_printer.py
M tests/shell/test_shell_commandline.py
9 files changed, 565 insertions(+), 147 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 17
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375: Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375: Impala shell outputs details of each RPC
......................................................................


Patch Set 14: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8956/


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 14
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Jan 2023 04:19:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................

IMPALA-11375 Impala shell outputs details of each RPC

When the Impala shell is using the hs2 protocol, it makes multiple RPCs
to the Impala daemon.  These calls pass Thrift objects back and forth.
This change adds the '--show_rpc' which outputs the details of the RPCs
to stdout and the '--rpc_file' flag which outputs the RPC details to the
specified file path.

RPC details include:
- operation name
- request attempt count
- Impala session/query ids (if applicable)
- call duration
- call status (success/failure)
- request Thrift objects
- response Thrift objects

Certain information is not included in the RPC details:
- Thrift object attributes named 'secret' or 'password'
  are redacted.
- Thrift objects with a type of TRowSet or TGetRuntimeProfileResp
  are not include as the information contained within them is
  already available in the standard output from the Impala shell.

Testing:
- Added new tests in the end-to-end test suite.

Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
---
M bin/impala-shell.sh
M shell/impala_client.py
M shell/impala_shell.py
M shell/impala_shell_config_defaults.py
M shell/make_shell_tarball.sh
M shell/option_parser.py
M shell/packaging/make_python_package.sh
A shell/thrift_printer.py
M tests/shell/test_shell_commandline.py
9 files changed, 556 insertions(+), 147 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/11
-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 11
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................

IMPALA-11375 Impala shell outputs details of each RPC

When the Impala shell is using the hs2 protocol, it makes multiple RPCs
to the Impala daemon.  These calls pass Thrift objects back and forth.
This change adds the '--show_rpc' which outputs the details of the RPCs
to stdout and the '--rpc_file' flag which outputs the RPC details to the
specified file path.

RPC details include:
- operation name
- request attempt count
- Impala session/query ids (if applicable)
- call duration
- call status (success/failure)
- request Thrift objects
- response Thrift objects

Certain information is not included in the RPC details:
- Thrift object attributes named 'secret' or 'password'
  are redacted.
- Thrift objects with a type of TRowSet or TGetRuntimeProfileResp
  are not include as the information contained within them is
  already available in the standars output from the Impala shell.

Testing:
- Added new tests in the end-to-end test suite.

Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
---
M bin/impala-shell.sh
M shell/impala_client.py
M shell/impala_shell.py
M shell/impala_shell_config_defaults.py
M shell/make_shell_tarball.sh
M shell/option_parser.py
M shell/packaging/make_python_package.sh
A shell/thrift_printer.py
M tests/shell/test_shell_commandline.py
9 files changed, 554 insertions(+), 150 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19388/8
-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-11375 Impala shell outputs details of each RPC

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19388 )

Change subject: IMPALA-11375 Impala shell outputs details of each RPC
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/12096/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19388
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I36f8dbc96726aa2a573133acbe8a558299381f8b
Gerrit-Change-Number: 19388
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Jan 2023 16:54:51 +0000
Gerrit-HasComments: No