You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Steve Carlin (Code Review)" <ge...@cloudera.org> on 2022/03/30 22:58:11 UTC
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Steve Carlin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18370
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
The strict hs2 protocol mode is broken when fetching large results.
The FetchResults.hasMoreRows field is always returned as false. When
there are no more results, Hive returns an empty batch with no rows.
Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
---
M shell/impala_client.py
M tests/shell/test_shell_commandline.py
2 files changed, 13 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/1
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 1
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18370/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18370/1//COMMIT_MSG@12
PS1, Line 12:
nit: add a sentence about what type of test has been added.
http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py@1126
PS1, Line 1126: tcol = col_value_converters[0][0](resp.results.columns[0])
Is checking the number of columns the safest way ? Is there an alternative to simply check the number of rows such that we don't have to dereference a column array ? Or perhaps if this pattern is used elsewhere here let me know.
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 1
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 18:19:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18370/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18370/1//COMMIT_MSG@12
PS1, Line 12:
> nit: add a sentence about what type of test has been added.
Done
http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py@1126
PS1, Line 1126: tcol = col_value_converters[0][0](resp.results.columns[0])
> Is checking the number of columns the safest way ? Is there an alternative
Yeah, this is the only way as far as I can tell.
I looked at Hive Beeline implementation and they iterate through the columns. There is no field which provides the number of rows. The columns are the only place where we can find this information.
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 1
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 21:30:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8000/
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 04:53:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG@10
PS2, Line 10: The FetchResults.hasMoreRows field is always returned as false. When
: there are no more results, Hive returns an empty batch with no rows.
> It's not clear to me - is this a bug in Hive? Is there a ticket for it?
Heh, I thought about this too...
The thing is...since the thrift protocol is defined by Hive, Hive is the one that really has the final say on how things are returned.
I discussed offline with a couple of people about potentially changing the server side. But right now, Hive hardcodes the "False" value into hasMoreRows. We could change that, but that has the potential (however small) of creating backward compatibility issues.
Is it a bug? Well...kinda. Why have the field if you're not gonna use it? But as I mentioned, their "beeline" code clearly does not make use of this field.
I just filed HIVE-26108 for this and logged it here.
http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG@15
PS2, Line 15: enough to ensure that multiple fetches are done.
> nit: extra line before change-id
Done
http://gerrit.cloudera.org:8080/#/c/18370/2/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/18370/2/shell/impala_client.py@1127
PS2, Line 1127: values
> Can't this be empty if all values are null? In Impala it can't, I am not su
In order for this to happen, Hive would have to be smart enough to realize that all values are null ahead of time before processing which seems very unlikely.
Having said that, unlikely is not "never", so I suppose it makes sense to add this.
http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py@1100
PS2, Line 1100: 1000000
> Do we need to run such a large query? Wouldn't it be enough to have more
I debated with myself over the number to choose. I chose an arbitrarily high number because the default can theoretically change and I didn't want to lose this test if that happened.
Having said that, I'm open to lowering it a little bit, I'll change it to 100,000
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 14:04:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8002/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 05:02:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG@10
PS2, Line 10: The FetchResults.hasMoreRows field is always returned as false. When
: there are no more results, Hive returns an empty batch with no rows.
It's not clear to me - is this a bug in Hive? Is there a ticket for it?
http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG@15
PS2, Line 15: enough to ensure that multiple fetches are done.
nit: extra line before change-id
http://gerrit.cloudera.org:8080/#/c/18370/2/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/18370/2/shell/impala_client.py@1127
PS2, Line 1127: values
Can't this be empty if all values are null? In Impala it can't, I am not sure about Hive. It may make sense to add a test for this (setting fetch size to 1 and run a query with multiple lines where the first line is always NULL
http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py@1100
PS2, Line 1100: 1000000
Do we need to run such a large query? Wouldn't it be enough to have more rows than fetch size?
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 12:07:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 6: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 22:03:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10366/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 1
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 23:18:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 09:32:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10379/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 14:11:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18370
to look at the new patch set (#5).
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
The strict hs2 protocol mode is broken when fetching large results.
The FetchResults.hasMoreRows field is always returned as false. When
there are no more results, Hive returns an empty batch with no rows.
HIVE-26108 has been filed to support the hasMoreRows field.
Added a framework test that retrieves 1M rows from tpcds. The default
number of rows returned from Hive is 10K so this should be more than
enough to ensure that multiple fetches are done.
Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
---
M shell/impala_client.py
M tests/shell/test_shell_commandline.py
2 files changed, 19 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/5
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
The strict hs2 protocol mode is broken when fetching large results.
The FetchResults.hasMoreRows field is always returned as false. When
there are no more results, Hive returns an empty batch with no rows.
HIVE-26108 has been filed to support the hasMoreRows field.
Added a framework test that retrieves 1M rows from tpcds. The default
number of rows returned from Hive is 10K so this should be more than
enough to ensure that multiple fetches are done.
Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Reviewed-on: http://gerrit.cloudera.org:8080/18370
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
---
M shell/impala_client.py
M tests/shell/test_shell_commandline.py
2 files changed, 19 insertions(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Verified
Wenzhe Zhou: Looks good to me, but someone else must approve
Csaba Ringhofer: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7986/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 1
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 23:21:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8005/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 17:30:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 1: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7986/
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 1
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 31 Mar 2022 03:49:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18370
to look at the new patch set (#2).
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
The strict hs2 protocol mode is broken when fetching large results.
The FetchResults.hasMoreRows field is always returned as false. When
there are no more results, Hive returns an empty batch with no rows.
Added a framework test that retrieves 1M rows from tpcds. The default
number of rows returned from Hive is 10K so this should be more than
enough to ensure that multiple fetches are done.
Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
---
M shell/impala_client.py
M tests/shell/test_shell_commandline.py
2 files changed, 13 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/2
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 2: Code-Review+2
(1 comment)
Carry Wenzhe's +1 and add mine and bumping to +2
http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/18370/1/shell/impala_client.py@1126
PS1, Line 1126: tcol = col_value_converters[0][0](resp.results.columns[0])
> Yeah, this is the only way as far as I can tell.
In that case, this should be good.
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 00:30:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 00:21:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8000/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 2
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 00:23:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 6: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 22:04:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 6: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/18370/2//COMMIT_MSG@10
PS2, Line 10: The FetchResults.hasMoreRows field is always returned as false. When
: there are no more results, Hive returns an empty batch with no rows.
> Heh, I thought about this too...
hasMoreRows is an optional member, so the server could decide to not set it at all. If someone sets it, then it should really mean that there are no more rows.
http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/18370/2/tests/shell/test_shell_commandline.py@1100
PS2, Line 1100:
> I debated with myself over the number to choose. I chose an arbitrarily hi
I think it should be enough to simply set --fetch_size to a very small value - as Hive always sets hasMoreRows, we simply need more then one fetch RPCs from impala-shell
(my concern is not about the size of the query from Impala perspective, but about the time needed to handle the answer in the client)
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 6
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Sat, 02 Apr 2022 08:08:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18370
to look at the new patch set (#4).
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
The strict hs2 protocol mode is broken when fetching large results.
The FetchResults.hasMoreRows field is always returned as false. When
there are no more results, Hive returns an empty batch with no rows.
HIVE-26108 has been filed to support the hasMoreRows field.
Added a framework test that retrieves 1M rows from tpcds. The default
number of rows returned from Hive is 10K so this should be more than
enough to ensure that multiple fetches are done.
Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
---
M shell/impala_client.py
M tests/shell/test_shell_commandline.py
2 files changed, 19 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/4
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18370
to look at the new patch set (#3).
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
The strict hs2 protocol mode is broken when fetching large results.
The FetchResults.hasMoreRows field is always returned as false. When
there are no more results, Hive returns an empty batch with no rows.
Added a framework test that retrieves 1M rows from tpcds. The default
number of rows returned from Hive is 10K so this should be more than
enough to ensure that multiple fetches are done.
Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
---
M shell/impala_client.py
M tests/shell/test_shell_commandline.py
2 files changed, 19 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/18370/3
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18370 )
Change subject: IMPALA-11213: Fixed impala-shell strict hs2 mode for large fetches
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10381/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18370
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife436d91e7fe0c30bf020024e20a5d8ad89faa24
Gerrit-Change-Number: 18370
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 14:23:32 +0000
Gerrit-HasComments: No