You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2022/03/17 02:49:42 UTC
[Impala-ASF-CR] IMPALA-11192: Batch uploading files in test scanner fuzz.py
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18329
Change subject: IMPALA-11192: Batch uploading files in test_scanner_fuzz.py
......................................................................
IMPALA-11192: Batch uploading files in test_scanner_fuzz.py
test_scanner_fuzz.py runs much slower on ORC than other formats. The
majority of the time is spent in uploading local files one by one to the
hdfs table folder.
The local files are copied from hdfs and randomly corrupted by the test.
The directory layout remains the same as the table folder. There are no
staging dirs that we should skip. So we can upload the whole local
folder at once, which saves a lot of the test time.
Tested locally and verified profiles of the succeeded queries. They all
scan the expected number of rows.
Change-Id: I504e160b84b3cc01d3be0b4e242d3c372692d181
---
M tests/query_test/test_scanners_fuzz.py
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/18329/1
--
To view, visit http://gerrit.cloudera.org:8080/18329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I504e160b84b3cc01d3be0b4e242d3c372692d181
Gerrit-Change-Number: 18329
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11192: Batch uploading files in test scanner fuzz.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18329 )
Change subject: IMPALA-11192: Batch uploading files in test_scanner_fuzz.py
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I504e160b84b3cc01d3be0b4e242d3c372692d181
Gerrit-Change-Number: 18329
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Mar 2022 06:44:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11192: Batch uploading files in test scanner fuzz.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18329 )
Change subject: IMPALA-11192: Batch uploading files in test_scanner_fuzz.py
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7953/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I504e160b84b3cc01d3be0b4e242d3c372692d181
Gerrit-Change-Number: 18329
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Mar 2022 06:44:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11192: Batch uploading files in test scanner fuzz.py
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18329 )
Change subject: IMPALA-11192: Batch uploading files in test_scanner_fuzz.py
......................................................................
Patch Set 1: Code-Review+2
Thanks for fixing this!
--
To view, visit http://gerrit.cloudera.org:8080/18329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I504e160b84b3cc01d3be0b4e242d3c372692d181
Gerrit-Change-Number: 18329
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 21 Mar 2022 09:05:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11192: Batch uploading files in test scanner fuzz.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18329 )
Change subject: IMPALA-11192: Batch uploading files in test_scanner_fuzz.py
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10300/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I504e160b84b3cc01d3be0b4e242d3c372692d181
Gerrit-Change-Number: 18329
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 17 Mar 2022 03:09:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11192: Batch uploading files in test scanner fuzz.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18329 )
Change subject: IMPALA-11192: Batch uploading files in test_scanner_fuzz.py
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I504e160b84b3cc01d3be0b4e242d3c372692d181
Gerrit-Change-Number: 18329
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Mar 2022 11:10:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11192: Batch uploading files in test scanner fuzz.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18329 )
Change subject: IMPALA-11192: Batch uploading files in test_scanner_fuzz.py
......................................................................
IMPALA-11192: Batch uploading files in test_scanner_fuzz.py
test_scanner_fuzz.py runs much slower on ORC than other formats. The
majority of the time is spent in uploading local files one by one to the
hdfs table folder.
The local files are copied from hdfs and randomly corrupted by the test.
The directory layout remains the same as the table folder. There are no
staging dirs that we should skip. So we can upload the whole local
folder at once, which saves a lot of the test time.
Tested locally and verified profiles of the succeeded queries. They all
scan the expected number of rows.
Change-Id: I504e160b84b3cc01d3be0b4e242d3c372692d181
Reviewed-on: http://gerrit.cloudera.org:8080/18329
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M tests/query_test/test_scanners_fuzz.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/18329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I504e160b84b3cc01d3be0b4e242d3c372692d181
Gerrit-Change-Number: 18329
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>