You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Vladimir Ozerov <vo...@gridgain.com> on 2016/12/16 15:23:15 UTC
Interesting statistics for our tests
Igniters,
I collected several thought-provoking numbers for our TeamCity
(config-variations tests are excluded):
* Number of Java suites: 78
* Total execution time without build overhead: 38.5 hours
* Total test classes: 3376
* Total tests: 60681
* 25% of time is consumed by 881 tests (41 test class)
* 50% of time is consumed by 2755 tests (153 test class)
* 75% of time is consumed by 15911 tests (443 test class)
Or to be more demonstrative:
* 1.5% of tests consume 25% of time
* 4.5% of tests consume 50% of time
* 26.2% tests consume 75% of time
TOP 3 test classes:
* CacheSerializableTransactionsTest - 46 minutes
* IgniteCacheCrossCacheTxFailoverTest - 30 minutes
* GridCachePartitionedNodeRestartTest - 25 minutes
Morale:
1) We have lots of poor failover/concurrency tests requiring
disproportional time to run;
2) If we split all tests into "core" and "failover/concurrency" parts
properly, then *core* part will take *~15 mins* to answer you with high
level of confidence whether your change broken anything or not.
Thoughts?
Vladimir.
Re: Interesting statistics for our tests
Posted by Yakov Zhdanov <yz...@apache.org>.
Agree with Vladimir's suggestions. Great analysis!
--Yakov
2016-12-17 11:36 GMT+03:00 Vladimir Ozerov <vo...@gridgain.com>:
> Dima,
> I do not think we really need more hardware for now. 40 min test will not
> run faster if we add more agents. We have a problem with tests quality, not
> quantity.
>
> Rohit,
> Yes, Ignite is ver advanced framework, so we have about 80-90 different
> suites, testing different parts of API.
>
> Vladimir.
>
> On Sat, Dec 17, 2016 at 2:47 AM, Rohit Mohta <rm...@gmail.com>
> wrote:
>
> > Interesting numbers, to say the least :)
> >
> > Separating test suites, as suggested by Vladimir sounds good.
> >
> > Probably off-placed question:
> > As Ignite provides multiple features and is suitable for multiple use
> > case- should we categorize test suites based on features?
> >
> > ------
> > Rohit Mohta
> >
> > > On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <ds...@apache.org>
> > wrote:
> > >
> > > In addition to optimizing the test suites, should we add more TC agents
> > as
> > > well?
> > >
> > > On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <vozerov@gridgain.com
> >
> > > wrote:
> > >
> > >> And one more thing - how much time is spent on duplicated "Binary"
> > suites?
> > >> *15
> > >> hours*, or almost *40%* of total "Run All" execution time. It means,
> > that
> > >> if move suites for *OptimizedMarshaller *(which we consider
> deprecated)
> > >> into a separate circuit (which will be executed not per-branch, but
> e.g.
> > >> nightly) we will almost double peak TeamCity throughput.
> > >>
> > >> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <
> vozerov@gridgain.com>
> > >> wrote:
> > >>
> > >>> Igniters,
> > >>>
> > >>> I collected several thought-provoking numbers for our TeamCity
> > >>> (config-variations tests are excluded):
> > >>>
> > >>> * Number of Java suites: 78
> > >>> * Total execution time without build overhead: 38.5 hours
> > >>> * Total test classes: 3376
> > >>> * Total tests: 60681
> > >>>
> > >>> * 25% of time is consumed by 881 tests (41 test class)
> > >>> * 50% of time is consumed by 2755 tests (153 test class)
> > >>> * 75% of time is consumed by 15911 tests (443 test class)
> > >>>
> > >>> Or to be more demonstrative:
> > >>> * 1.5% of tests consume 25% of time
> > >>> * 4.5% of tests consume 50% of time
> > >>> * 26.2% tests consume 75% of time
> > >>>
> > >>> TOP 3 test classes:
> > >>> * CacheSerializableTransactionsTest - 46 minutes
> > >>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
> > >>> * GridCachePartitionedNodeRestartTest - 25 minutes
> > >>>
> > >>> Morale:
> > >>> 1) We have lots of poor failover/concurrency tests requiring
> > >>> disproportional time to run;
> > >>> 2) If we split all tests into "core" and "failover/concurrency" parts
> > >>> properly, then *core* part will take *~15 mins* to answer you with
> high
> > >>> level of confidence whether your change broken anything or not.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> Vladimir.
> > >>>
> > >>
> >
>
Re: Interesting statistics for our tests
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Dima,
I do not think we really need more hardware for now. 40 min test will not
run faster if we add more agents. We have a problem with tests quality, not
quantity.
Rohit,
Yes, Ignite is ver advanced framework, so we have about 80-90 different
suites, testing different parts of API.
Vladimir.
On Sat, Dec 17, 2016 at 2:47 AM, Rohit Mohta <rm...@gmail.com> wrote:
> Interesting numbers, to say the least :)
>
> Separating test suites, as suggested by Vladimir sounds good.
>
> Probably off-placed question:
> As Ignite provides multiple features and is suitable for multiple use
> case- should we categorize test suites based on features?
>
> ------
> Rohit Mohta
>
> > On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
> >
> > In addition to optimizing the test suites, should we add more TC agents
> as
> > well?
> >
> > On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <vo...@gridgain.com>
> > wrote:
> >
> >> And one more thing - how much time is spent on duplicated "Binary"
> suites?
> >> *15
> >> hours*, or almost *40%* of total "Run All" execution time. It means,
> that
> >> if move suites for *OptimizedMarshaller *(which we consider deprecated)
> >> into a separate circuit (which will be executed not per-branch, but e.g.
> >> nightly) we will almost double peak TeamCity throughput.
> >>
> >> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <vo...@gridgain.com>
> >> wrote:
> >>
> >>> Igniters,
> >>>
> >>> I collected several thought-provoking numbers for our TeamCity
> >>> (config-variations tests are excluded):
> >>>
> >>> * Number of Java suites: 78
> >>> * Total execution time without build overhead: 38.5 hours
> >>> * Total test classes: 3376
> >>> * Total tests: 60681
> >>>
> >>> * 25% of time is consumed by 881 tests (41 test class)
> >>> * 50% of time is consumed by 2755 tests (153 test class)
> >>> * 75% of time is consumed by 15911 tests (443 test class)
> >>>
> >>> Or to be more demonstrative:
> >>> * 1.5% of tests consume 25% of time
> >>> * 4.5% of tests consume 50% of time
> >>> * 26.2% tests consume 75% of time
> >>>
> >>> TOP 3 test classes:
> >>> * CacheSerializableTransactionsTest - 46 minutes
> >>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
> >>> * GridCachePartitionedNodeRestartTest - 25 minutes
> >>>
> >>> Morale:
> >>> 1) We have lots of poor failover/concurrency tests requiring
> >>> disproportional time to run;
> >>> 2) If we split all tests into "core" and "failover/concurrency" parts
> >>> properly, then *core* part will take *~15 mins* to answer you with high
> >>> level of confidence whether your change broken anything or not.
> >>>
> >>> Thoughts?
> >>>
> >>> Vladimir.
> >>>
> >>
>
Re: Interesting statistics for our tests
Posted by Rohit Mohta <rm...@gmail.com>.
Interesting numbers, to say the least :)
Separating test suites, as suggested by Vladimir sounds good.
Probably off-placed question:
As Ignite provides multiple features and is suitable for multiple use case- should we categorize test suites based on features?
------
Rohit Mohta
> On Dec 16, 2016, at 14:04, Dmitriy Setrakyan <ds...@apache.org> wrote:
>
> In addition to optimizing the test suites, should we add more TC agents as
> well?
>
> On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
>> And one more thing - how much time is spent on duplicated "Binary" suites?
>> *15
>> hours*, or almost *40%* of total "Run All" execution time. It means, that
>> if move suites for *OptimizedMarshaller *(which we consider deprecated)
>> into a separate circuit (which will be executed not per-branch, but e.g.
>> nightly) we will almost double peak TeamCity throughput.
>>
>> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <vo...@gridgain.com>
>> wrote:
>>
>>> Igniters,
>>>
>>> I collected several thought-provoking numbers for our TeamCity
>>> (config-variations tests are excluded):
>>>
>>> * Number of Java suites: 78
>>> * Total execution time without build overhead: 38.5 hours
>>> * Total test classes: 3376
>>> * Total tests: 60681
>>>
>>> * 25% of time is consumed by 881 tests (41 test class)
>>> * 50% of time is consumed by 2755 tests (153 test class)
>>> * 75% of time is consumed by 15911 tests (443 test class)
>>>
>>> Or to be more demonstrative:
>>> * 1.5% of tests consume 25% of time
>>> * 4.5% of tests consume 50% of time
>>> * 26.2% tests consume 75% of time
>>>
>>> TOP 3 test classes:
>>> * CacheSerializableTransactionsTest - 46 minutes
>>> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
>>> * GridCachePartitionedNodeRestartTest - 25 minutes
>>>
>>> Morale:
>>> 1) We have lots of poor failover/concurrency tests requiring
>>> disproportional time to run;
>>> 2) If we split all tests into "core" and "failover/concurrency" parts
>>> properly, then *core* part will take *~15 mins* to answer you with high
>>> level of confidence whether your change broken anything or not.
>>>
>>> Thoughts?
>>>
>>> Vladimir.
>>>
>>
Re: Interesting statistics for our tests
Posted by Dmitriy Setrakyan <ds...@apache.org>.
In addition to optimizing the test suites, should we add more TC agents as
well?
On Fri, Dec 16, 2016 at 7:51 AM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> And one more thing - how much time is spent on duplicated "Binary" suites?
> *15
> hours*, or almost *40%* of total "Run All" execution time. It means, that
> if move suites for *OptimizedMarshaller *(which we consider deprecated)
> into a separate circuit (which will be executed not per-branch, but e.g.
> nightly) we will almost double peak TeamCity throughput.
>
> On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <vo...@gridgain.com>
> wrote:
>
> > Igniters,
> >
> > I collected several thought-provoking numbers for our TeamCity
> > (config-variations tests are excluded):
> >
> > * Number of Java suites: 78
> > * Total execution time without build overhead: 38.5 hours
> > * Total test classes: 3376
> > * Total tests: 60681
> >
> > * 25% of time is consumed by 881 tests (41 test class)
> > * 50% of time is consumed by 2755 tests (153 test class)
> > * 75% of time is consumed by 15911 tests (443 test class)
> >
> > Or to be more demonstrative:
> > * 1.5% of tests consume 25% of time
> > * 4.5% of tests consume 50% of time
> > * 26.2% tests consume 75% of time
> >
> > TOP 3 test classes:
> > * CacheSerializableTransactionsTest - 46 minutes
> > * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
> > * GridCachePartitionedNodeRestartTest - 25 minutes
> >
> > Morale:
> > 1) We have lots of poor failover/concurrency tests requiring
> > disproportional time to run;
> > 2) If we split all tests into "core" and "failover/concurrency" parts
> > properly, then *core* part will take *~15 mins* to answer you with high
> > level of confidence whether your change broken anything or not.
> >
> > Thoughts?
> >
> > Vladimir.
> >
>
Re: Interesting statistics for our tests
Posted by Vladimir Ozerov <vo...@gridgain.com>.
And one more thing - how much time is spent on duplicated "Binary" suites? *15
hours*, or almost *40%* of total "Run All" execution time. It means, that
if move suites for *OptimizedMarshaller *(which we consider deprecated)
into a separate circuit (which will be executed not per-branch, but e.g.
nightly) we will almost double peak TeamCity throughput.
On Fri, Dec 16, 2016 at 6:23 PM, Vladimir Ozerov <vo...@gridgain.com>
wrote:
> Igniters,
>
> I collected several thought-provoking numbers for our TeamCity
> (config-variations tests are excluded):
>
> * Number of Java suites: 78
> * Total execution time without build overhead: 38.5 hours
> * Total test classes: 3376
> * Total tests: 60681
>
> * 25% of time is consumed by 881 tests (41 test class)
> * 50% of time is consumed by 2755 tests (153 test class)
> * 75% of time is consumed by 15911 tests (443 test class)
>
> Or to be more demonstrative:
> * 1.5% of tests consume 25% of time
> * 4.5% of tests consume 50% of time
> * 26.2% tests consume 75% of time
>
> TOP 3 test classes:
> * CacheSerializableTransactionsTest - 46 minutes
> * IgniteCacheCrossCacheTxFailoverTest - 30 minutes
> * GridCachePartitionedNodeRestartTest - 25 minutes
>
> Morale:
> 1) We have lots of poor failover/concurrency tests requiring
> disproportional time to run;
> 2) If we split all tests into "core" and "failover/concurrency" parts
> properly, then *core* part will take *~15 mins* to answer you with high
> level of confidence whether your change broken anything or not.
>
> Thoughts?
>
> Vladimir.
>