[jira] [Resolved] (IMPALA-6372) Dataload should execute Hive loads in parallel

["Joe McDonnell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joe McDonnell resolved IMPALA-6372.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 3.0

> Dataload should execute Hive loads in parallel
> ----------------------------------------------
>
>                 Key: IMPALA-6372
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6372
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Infrastructure
>    Affects Versions: Impala 2.12.0
>            Reporter: Joe McDonnell
>            Assignee: Joe McDonnell
>            Priority: Major
>             Fix For: Impala 3.0
>
>
> Currently, bin/load-data.py goes parallel on parts of the Impala DDLs and Impala inserts (for Parquet), but it does not go parallel on the Hive portion of dataload. testdata/bin/generate-schema-statements.py only generates a single Hive file that load-data.py executes serially.
> generate-schema-statements.py should generate multiple SQL files for the Hive load portion and load-data.py should execute them in parallel (once the base text tables have been loaded).
> Even with parallel execution of TPC-H, functional, and TPC-DS, this will still deliver speedups for dataload (and GVO).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)