[jira] [Created] (IMPALA-15185) Increase Hive parallelism during dataload

["Csaba Ringhofer (Jira)" <ji...@apache.org>]

Csaba Ringhofer created IMPALA-15185:
----------------------------------------

             Summary: Increase Hive parallelism during dataload
                 Key: IMPALA-15185
                 URL: https://issues.apache.org/jira/browse/IMPALA-15185
             Project: IMPALA
          Issue Type: Improvement
            Reporter: Csaba Ringhofer


The combination of different configs limit the number of parallel Hive queries to 2:
{code}
'yarn.scheduler.minimum-allocation-mb': 2048
{code}
{code}
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.1</value>
{code}
{code}
 ret = min(max(available_ram_gb * 1024 - 20 * 1024, 4096), 48 * 1024)
 print("Configuring Yarn NM to use {0}MB RAM".format(ret), file=sys.stderr)
{code}

On typical 64GB machines:
44GB for Yarn
4.4GB for AMs (0.1: 10% of total)
2*2GB (min 2GB for each yarn container)
40 GB (rest of yarn mem) for Tez / MR workers

The typical queries during dataload are pretty small and don't need 20 workers per AM.
The current setup also seems to play a role in the very slow Hive ACID compactions (don't understand how exactly).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)