You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2026/07/08 20:57:00 UTC
[jira] [Commented] (IMPALA-14697) Generate TPC-DS / TPC-H datasets in parallel
[ https://issues.apache.org/jira/browse/IMPALA-14697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094683#comment-18094683 ]
ASF subversion and git services commented on IMPALA-14697:
----------------------------------------------------------
Commit 770bf8464416506811af312858268828b0cef5dc in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=770bf8464 ]
IMPALA-14697: Generate TPC-H/TPC-DS data in parallel
For performance test jobs, generating the datasets at higher
scales can take significant time. Currently, this is running
a single invocation of the generator binary in a single
thread. Both TPC-H and TPC-DS have an ability to generate data
in parallel, producing multiple files for larger tables. This
keeps the current behavior for scale factor 1 to avoid having
a different number of files. This generates higher scale
factors with $(nproc) parallel threads. The LOAD DATA statements
for these tables already reference the directory name itself,
so it doesn't matter that there are more files.
Testing:
- Ran core job
- Loaded TPC-H scale 42
- Loaded TPC-DS scale 20
Change-Id: I7ba13e2275be2ac1a5ae8f9354c947d9f1adf263
Reviewed-on: http://gerrit.cloudera.org:8080/23092
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Generate TPC-DS / TPC-H datasets in parallel
> --------------------------------------------
>
> Key: IMPALA-14697
> URL: https://issues.apache.org/jira/browse/IMPALA-14697
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
>
> For performance test jobs, generating TPC-H or TPC-DS at higher scales can take noticeable time (several minutes). Currently, a single call to the generator is producing all the tables in a single thread. This can be parallelized by having multiple threads each generate the data for a single table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org