You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Sahil Takiar (JIRA)" <ji...@apache.org> on 2018/06/06 22:38:00 UTC
[jira] [Commented] (HIVE-19571) Ability to run multiple pre-commit
jobs on a ptest server
[ https://issues.apache.org/jira/browse/HIVE-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504003#comment-16504003 ]
Sahil Takiar commented on HIVE-19571:
-------------------------------------
Moving this to unassigned as I won't be able to work on this anytime soon. Anyone else feel free to pick this up.
I think the attached patch solves a lot of the issues, but it still needs to be tested. We might be able to get away with just writing Java unit tests to test this.
> Ability to run multiple pre-commit jobs on a ptest server
> ---------------------------------------------------------
>
> Key: HIVE-19571
> URL: https://issues.apache.org/jira/browse/HIVE-19571
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-17317.WIP.1.patch
>
>
> I've been taking a look at the Disk, Network, and CPU usage of the GCE instances we run ptest on, and it doesn't look like we are fully utilizing the machines. The resource usage is very up and down.
> During each ptest execution, there is a large chunk of time (~20 min) where its just the Jenkins job that is doing any work (checking out github repos, building code, figuring out test batches, etc.). During this time, the ptest nodes are mostly idle - the CPU and Disk I/O are almost zero.
> Even when ptest is running, I think some of resources are under-utilized. Network and disk resource spike at the beginning of the job, probably because ptest is distributing resources to each machine, each slave is downloading jars, etc. However, after that, when the actual tests run, there is almost 0 network activity (which makes sense since tests runs on a single node). For disk usage, there is activity, but not nearly as high as when the setup phase was occuring. CPU usage fluctuates between 40-80%.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)