[jira] [Created] (HIVE-19571) Ability to run multiple pre-commit jobs on a ptest server

["Sahil Takiar (JIRA)" <ji...@apache.org>]

Sahil Takiar created HIVE-19571:
-----------------------------------

             Summary: Ability to run multiple pre-commit jobs on a ptest server
                 Key: HIVE-19571
                 URL: https://issues.apache.org/jira/browse/HIVE-19571
             Project: Hive
          Issue Type: Sub-task
          Components: Testing Infrastructure
            Reporter: Sahil Takiar
            Assignee: Sahil Takiar


I've been taking a look at the Disk, Network, and CPU usage of the GCE instances we run ptest on, and it doesn't look like we are fully utilizing the machines. The resource usage is very up and down.

During each ptest execution, there is a large chunk of time (~20 min) where its just the Jenkins job that is doing any work (checking out github repos, building code, figuring out test batches, etc.). During this time, the ptest nodes are mostly idle - the CPU and Disk I/O are almost zero.

Even when ptest is running, I think some of resources are under-utilized. Network and disk resource spike at the beginning of the job, probably because ptest is distributing resources to each machine, each slave is downloading jars, etc. However, after that, when the actual tests run, there is almost 0 network activity (which makes sense since tests runs on a single node). For disk usage, there is activity, but not nearly as high as when the setup phase was occuring. CPU usage fluctuates between 40-80%.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)