You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Philip Zeyliger (Code Review)" <ge...@cloudera.org> on 2017/12/13 22:27:56 UTC
[Impala-ASF-CR] IMPALA-6070: Parallelize another bit of data load.
Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8822
Change subject: IMPALA-6070: Parallelize another bit of data load.
......................................................................
IMPALA-6070: Parallelize another bit of data load.
The two Kudu loads and Hive UDFs can all run in parallel. This
should shave about 4 minutes off of the data load. (Current
timings are 3.5, 4, and 0.6 minutes, see below.)
I've run dataload with this change many times.
Loading Kudu functional (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu.log)...
Loading workload 'functional-query' using exploration strategy 'core' in table formats 'kudu/none/none' OK (Took: 3 min 29 sec)
Loading Kudu TPCH (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu-tpch.log)...
Loading workload 'tpch' using exploration strategy 'core' in table formats 'kudu/none/none' OK (Took: 4 min 0 sec)
Loading Hive UDFs (logging to /home/ubuntu/Impala/logs/data_loading/build-and-copy-hive-udfs.log)...
Loading Hive UDFs OK (Took: 0 min 41 sec)
Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
---
M testdata/bin/create-load-data.sh
1 file changed, 4 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/8822/1
--
To view, visit http://gerrit.cloudera.org:8080/8822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
Gerrit-Change-Number: 8822
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
[Impala-ASF-CR] IMPALA-6070: Parallelize another bit of data load.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8822 )
Change subject: IMPALA-6070: Parallelize another bit of data load.
......................................................................
IMPALA-6070: Parallelize another bit of data load.
The two Kudu loads and Hive UDFs can all run in parallel. This
should shave about 4 minutes off of the data load. (Current
timings are 3.5, 4, and 0.6 minutes, see below.)
I've run dataload with this change many times.
Loading Kudu functional (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu.log)...
Loading workload 'functional-query' using exploration strategy 'core' in table formats 'kudu/none/none' OK (Took: 3 min 29 sec)
Loading Kudu TPCH (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu-tpch.log)...
Loading workload 'tpch' using exploration strategy 'core' in table formats 'kudu/none/none' OK (Took: 4 min 0 sec)
Loading Hive UDFs (logging to /home/ubuntu/Impala/logs/data_loading/build-and-copy-hive-udfs.log)...
Loading Hive UDFs OK (Took: 0 min 41 sec)
Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
Reviewed-on: http://gerrit.cloudera.org:8080/8822
Reviewed-by: Dimitris Tsirogiannis <dt...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M testdata/bin/create-load-data.sh
1 file changed, 4 insertions(+), 3 deletions(-)
Approvals:
Dimitris Tsirogiannis: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/8822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
Gerrit-Change-Number: 8822
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
[Impala-ASF-CR] IMPALA-6070: Parallelize another bit of data load.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8822 )
Change subject: IMPALA-6070: Parallelize another bit of data load.
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1616/
--
To view, visit http://gerrit.cloudera.org:8080/8822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
Gerrit-Change-Number: 8822
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2017 23:05:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6070: Parallelize another bit of data load.
Posted by "Dimitris Tsirogiannis (Code Review)" <ge...@cloudera.org>.
Dimitris Tsirogiannis has posted comments on this change. ( http://gerrit.cloudera.org:8080/8822 )
Change subject: IMPALA-6070: Parallelize another bit of data load.
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/8822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
Gerrit-Change-Number: 8822
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2017 23:04:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6070: Parallelize another bit of data load.
Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8822 )
Change subject: IMPALA-6070: Parallelize another bit of data load.
......................................................................
Patch Set 1:
https://jenkins.impala.io/view/Utility/job/pre-review-test/87/ passed just fine.
The logs from https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/798/consoleFull look fine to me.
00:53:42 Creating many block table OK (Took: 0 min 23 sec)
00:53:42 Started Loading Kudu functional in background; pid 19994.
00:53:42 Loading Kudu functional (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu.log)...
00:53:42 Started Loading Kudu TPCH in background; pid 19995.
00:53:42 Loading Kudu TPCH (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu-tpch.log)...
00:53:42 Started Loading Hive UDFs in background; pid 19996.
00:53:42 Loading Hive UDFs (logging to /home/ubuntu/Impala/logs/data_loading/build-and-copy-hive-udfs.log)...
00:54:24 Loading Hive UDFs OK (Took: 0 min 42 sec)
00:56:56 Loading workload 'functional-query' using exploration strategy 'core' in table formats 'kudu/none/none' OK (Took: 3 min 14 sec)
00:57:38 Loading workload 'tpch' using exploration strategy 'core' in table formats 'kudu/none/none' OK (Took: 3 min 56 sec)
00:57:38 Running custom post-load steps (logging to /home/ubuntu/Impala/logs/data_loading/custom-post-load-steps.log)...
Comparing to some arbitrary other run of this job (https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/796/consoleFull; see below) this seems to save 4 minutes. The overall run time change here is still within the run-to-run noise, but I think this will move the baseline down 3-4 minutes.
19:26:09 Loading Kudu functional (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu.log)...
19:29:38 Loading workload 'functional-query' using exploration strategy 'core' in table formats 'kudu/none/none' OK (Took: 3 min 29 sec)
19:29:38 Loading Kudu TPCH (logging to /home/ubuntu/Impala/logs/data_loading/load-kudu-tpch.log)...
19:33:38 Loading workload 'tpch' using exploration strategy 'core' in table formats 'kudu/none/none' OK (Took: 4 min 0 sec)
19:33:38 Loading Hive UDFs (logging to /home/ubuntu/Impala/logs/data_loading/build-and-copy-hive-udfs.log)...
19:34:19 Loading Hive UDFs OK (Took: 0 min 41 sec)
19:34:19 Running custom post-load steps (logging to /home/ubuntu/Impala/logs/data_loading/custom-post-load-steps.log)...
--
To view, visit http://gerrit.cloudera.org:8080/8822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
Gerrit-Change-Number: 8822
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2017 22:31:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6070: Parallelize another bit of data load.
Posted by "Dimitris Tsirogiannis (Code Review)" <ge...@cloudera.org>.
Dimitris Tsirogiannis has posted comments on this change. ( http://gerrit.cloudera.org:8080/8822 )
Change subject: IMPALA-6070: Parallelize another bit of data load.
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/8822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
Gerrit-Change-Number: 8822
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Dec 2017 23:04:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6070: Parallelize another bit of data load.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8822 )
Change subject: IMPALA-6070: Parallelize another bit of data load.
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/8822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7e93ee5a77ec9271b980b88bef7ad512ecbe0407
Gerrit-Change-Number: 8822
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2017 02:28:39 +0000
Gerrit-HasComments: No