Integration Test Slow Down

[Elliott Clark <ec...@apache.org>]

So I went and checked on the run times of Cloudera's internal integration
test jenkins runs, and I noticed that recently the
IntegrationTestBigLinkedList runs started running much slower.

With the slow monkey:
5 hr 58 min -> 11 hr

With the calm monkey:
4 hr 36 min -> 6 hr 24 min

IntegrationTestBulkLoad also has a slow down but it's much less dramatic

22 min -> 26 min

This slow down doesn't seem to manifest itself with TestLoadAndVerify at
all.  But it has been pretty consistent since the build containing the
following commits:

Changes
HBASE-9949 Fix the race condition between Compaction and StoreScanner.init
HBASE-9949 Addendum removes readpoint parameter for getScanner()
HBASE-9949 Revert due to TestHRegion failure
HBASE-9869 Optimize HConnectionManager#getCachedLocation
HBASE-9983 Lower the memory footprint of HRegionLocation
HBASE-9869 Optimize HConnectionManager#getCachedLocation - remove
SoftValueSortedMap


To me the first three commits look like they cancel each so that leaves the
last three commits as the most likely culprits.  Does anyone have a thought
on whey these would be causing problems with ITBLL ?

Re: Integration Test Slow Down

[Elliott Clark <ec...@apache.org>]

On Thu, Dec 5, 2013 at 3:27 PM, Devaraj Das <dd...@hortonworks.com> wrote:

> Elliott, do you know how much it takes without chaosmonkey


That's what the Calm monkey does.  So 4.5 hours to 6.3

Re: Integration Test Slow Down

[Devaraj Das <dd...@hortonworks.com>]

Elliott, do you know how much it takes without chaosmonkey (no failures at
all). That would be another good data point I think.


On Thu, Dec 5, 2013 at 2:52 PM, Elliott Clark <ec...@apache.org> wrote:

> So I went and checked on the run times of Cloudera's internal integration
> test jenkins runs, and I noticed that recently the
> IntegrationTestBigLinkedList runs started running much slower.
>
> With the slow monkey:
> 5 hr 58 min -> 11 hr
>
> With the calm monkey:
> 4 hr 36 min -> 6 hr 24 min
>
> IntegrationTestBulkLoad also has a slow down but it's much less dramatic
>
> 22 min -> 26 min
>
> This slow down doesn't seem to manifest itself with TestLoadAndVerify at
> all.  But it has been pretty consistent since the build containing the
> following commits:
>
> Changes
> HBASE-9949 Fix the race condition between Compaction and StoreScanner.init
> HBASE-9949 Addendum removes readpoint parameter for getScanner()
> HBASE-9949 Revert due to TestHRegion failure
> HBASE-9869 Optimize HConnectionManager#getCachedLocation
> HBASE-9983 Lower the memory footprint of HRegionLocation
> HBASE-9869 Optimize HConnectionManager#getCachedLocation - remove
> SoftValueSortedMap
>
>
> To me the first three commits look like they cancel each so that leaves the
> last three commits as the most likely culprits.  Does anyone have a thought
> on whey these would be causing problems with ITBLL ?
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.