You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Павлухин Иван <vo...@gmail.com> on 2019/05/01 05:34:22 UTC
Re: Brainstorm: Make TC Run All faster
Nikolay,
> I thought by "faster" we all meant time between One push RunAll button and one get results.
> What else definition are possible here?
But this "faster" depends on a current load. E.g. if we have a lot of
idle slaves then incresead parallelism will allow a build to complete
"faster". If we have many slaves busy than parallelism might "slow"
down an execution. I predict that you are mostly interested in average
metrics when TC has a build queue as it usually have on weekdays. If
so then aggregatin other related jobs into "Build Apache Ignite" might
speed execution of RunAll measured in wall clock time.
And also I would like to return to a point I already mentioned earlier
in other conversations. Our CI environment suites perfectly for an
agile way of working. I suppose that we could try different setups
almost freely and see if it gives a desired effect (and rollback if
no). I am ok to go with Vyacheslav proposal. But I would like to
observe some metrics showing that builds become running faster than
before. Can we do this?
вт, 30 апр. 2019 г. в 17:42, Dmitriy Pavlov <dp...@apache.org>:
>
> Hi, One more metric we can introduce is efficiency. Test run time/overall
> time. The goal of launching RunAll is to obtain tests results.
>
> I've added more statistics to be displayed in the TC Bot, so now you may
> click 'more' button near 'chain results' caption and you'll see sum of
> statistics for all chain or individual suite. Latest RunAll gives the
> following:
> Overall time 57h 52m 41.188s (
> - Build Net Time: 45h 10m 31.686s,
> - Tests: 38h 58m 30.8s,
> - Src. Update: 2h 29m 51.14s,
> - Artifacts Publishing: 26m 38.295s,
> - Dependencies Resolving: 9h 45m 1.809s,
> - Timeouts: 3h 54m 16.071s)
>
> Currently, Efficiency is around 67-72%
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 29 апр. 2019 г. в 12:57, Nikolay Izhikov <ni...@apache.org>:
>
> > Ivan,
> >
> > I thought by "faster" we all meant time between One push RunAll button and
> > one get results.
> > What else definition are possible here?
> >
> > В Пн, 29/04/2019 в 12:04 +0300, Павлухин Иван пишет:
> > > Nikolay,
> > >
> > > > Why we should imagine it?
> > >
> > > Only in order to understand what do we mean by "faster". My question was:
> > >
> > > 1st approach will take less agent time in sum. 2nd approach will
> > > complete faster in wall clock time. And your main concern is "total
> > > agent time". Am I right?
> > >
> > > пн, 29 апр. 2019 г. в 12:01, Nikolay Izhikov <ni...@apache.org>:
> > > >
> > > > > Let's imagine that we have an infinite number of agents
> > > >
> > > > Why we should imagine it?
> > > > We don't have infinite number of agents.
> > > >
> > > > And we have several concurrent Run All.
> > > >
> > > >
> > > > В Пн, 29/04/2019 в 11:50 +0300, Павлухин Иван пишет:
> > > > > Vyacheslav,
> > > > >
> > > > > I finally figured out that "faster" means "total agent time".
> > > > >
> > > > > Let's imagine that we have an infinite number of agents. And 2
> > approaches:
> > > > > 1. Uber "Build Apache Ignite" containing all checks.
> > > > > 2. Separate jobs for compilation, checkstyle and etc.
> > > > >
> > > > > 1st approach will take less agent time in sum. 2nd approach will
> > > > > complete faster in wall clock time. And your main concern is "total
> > > > > agent time". Am I right?
> > > > >
> > > > > пн, 29 апр. 2019 г. в 11:42, Vyacheslav Daradur <daradurvs@gmail.com
> > >:
> > > > > >
> > > > > > Hi, Ivan,
> > > > > >
> > > > > > We are in the thread "Make TC Run All faster", so the main benefit
> > is
> > > > > > to make TC faster :)
> > > > > >
> > > > > > Advantages:
> > > > > > - 1 TC agent required instead of 4;
> > > > > > - RunAll will be faster, in case of builds queue;
> > > > > >
> > > > > > Also, the "licenses" profile is included in the step of a release
> > > > > > build. I believe check-style also should be included.
> > > > > >
> > > > > > The generation of Javadocs is an optional step at preparing the
> > > > > > release, but its check on TC takes significant time in case of the
> > > > > > separate build.
> > > > > >
> > > > > > > > Returning to "Build Apache Ignite" it seems to me that
> > ideally, it can
> > > > > >
> > > > > > be hierarchical.
> > > > > >
> > > > > > I agree, all the checks may be set as a separate step in the
> > build's
> > > > > > configuration. This helps with the main problem I'm trying to
> > solve -
> > > > > > resolving of dependencies which takes the most time of the builds.
> > > > > >
> > > > > > On Mon, Apr 29, 2019 at 11:24 AM Павлухин Иван <
> > vololo100@gmail.com> wrote:
> > > > > > >
> > > > > > > Vyacheslav, Maxim,
> > > > > > >
> > > > > > > Can we once again outline what benefits aggregated "Build Apache
> > > > > > > Ignite" performing various checks has comparing to a modularized
> > > > > > > approach in which separate builds perform separate tasks?
> > > > > > >
> > > > > > > For example, modularized approach looks nice because it is
> > similar to
> > > > > > > good practices in software development where we separate
> > > > > > > responsibilities between different classes instead of
> > aggregating them
> > > > > > > into a single class. And as usual multiple classes works together
> > > > > > > coordinating by a class from upper level. So, in fact it is a
> > > > > > > hierarchical structure.
> > > > > > >
> > > > > > > Returning to "Build Apache Ignite" it seems to me that ideally
> > it can
> > > > > > > be hierarchical. There is a top level compilation (assembly?)
> > job but
> > > > > > > it is always clear what tasks does it perform (check style, check
> > > > > > > license and other subjobs).
> > > > > > >
> > > > > > > пт, 26 апр. 2019 г. в 17:06, Maxim Muzafarov <maxmuzaf@gmail.com
> > >:
> > > > > > > >
> > > > > > > > Folks,
> > > > > > > >
> > > > > > > > +1 for merging all these suites into the single one. All these
> > suites
> > > > > > > > (Build Apache Ignite, Javadoc, Licenses Header, Checkstyle)
> > required
> > > > > > > > to be `green` all the time. So, we can consider making them a
> > part of
> > > > > > > > build Apache Ignite procedure.
> > > > > > > >
> > > > > > > > Also, I'd suggest going deeper. We can try to merge `Licenses
> > Header`
> > > > > > > > into the `Code style checker` [1]. This will simplify the code
> > > > > > > > checking process.
> > > > > > > >
> > > > > > > > [1] http://checkstyle.sourceforge.net/config_header.html
> > > > > > > >
> > > > > > > > On Fri, 26 Apr 2019 at 13:17, Vyacheslav Daradur <
> > daradurvs@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Ivan, you are right, I meant to combine them into one.
> > > > > > > > >
> > > > > > > > > Here is a build [1], with enabled profiles (check-licenses,
> > > > > > > > > checkstyle) and check of javadoc to show the idea.
> > > > > > > > >
> > > > > > > > > Seems it takes ~15 minutes.
> > > > > > > > >
> > > > > > > > > [1]
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ExperimentalBuildApacheIgniteJavadocLicensesHeaderCheckstyle&branch_IgniteTests24Java8=
> > <default>;;
> > > > > > > > >
> > > > > > > > > On Fri, Apr 26, 2019 at 12:06 PM Павлухин Иван <
> > vololo100@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Vyacheslav,
> > > > > > > > > >
> > > > > > > > > > What do you mean by uniting?
> > > > > > > > > >
> > > > > > > > > > For me it looks like [Javadocs] and [Check Code Style] are
> > not so time
> > > > > > > > > > consuming comparing to tests, are not they? Do you suggest
> > to combine
> > > > > > > > > > mentioned 4 jobs into one? How long will it run in a such
> > case?
> > > > > > > > > >
> > > > > > > > > > чт, 25 апр. 2019 г. в 10:50, Vyacheslav Daradur <
> > daradurvs@gmail.com>:
> > > > > > > > > > >
> > > > > > > > > > > Hi Igniters,
> > > > > > > > > > >
> > > > > > > > > > > At the moment we have several separated test suites:
> > > > > > > > > > > * ~Build Apache Ignite~ _ ~10..20mins
> > > > > > > > > > > * [Javadocs] _ ~10mins
> > > > > > > > > > > * [Licenses Headers] _ ~1min
> > > > > > > > > > > * [Check Code Style] _ ~7min
> > > > > > > > > > > The most time of each build (except Licenses Headers) is
> > taken by
> > > > > > > > > > > dependency resolving.
> > > > > > > > > > >
> > > > > > > > > > > Their main goal is a check that the project is built
> > properly.
> > > > > > > > > > >
> > > > > > > > > > > Also, profiles of [Javadocs], [Licenses Headers] uses at
> > the step of
> > > > > > > > > > > preparing release (see DEVNOTES.txt) that means they are
> > important.
> > > > > > > > > > >
> > > > > > > > > > > I'd suggest uniting the builds, this should reduce the
> > time of tests
> > > > > > > > > > > on ~15 minutes and releases agents.
> > > > > > > > > > >
> > > > > > > > > > > What do you think?
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 27, 2018 at 3:56 PM Павлухин Иван <
> > vololo100@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Roman,
> > > > > > > > > > > >
> > > > > > > > > > > > Do you have some expectations how faster "correlated"
> > tests
> > > > > > > > > > > > elimination will make Run All? Also do you have a
> > vision how can we
> > > > > > > > > > > > determine such "correlated" tests, can we do it
> > relatively fast?
> > > > > > > > > > > >
> > > > > > > > > > > > But all in all, I am not sure that reducing a group of
> > correlated
> > > > > > > > > > > > tests to only one test can show good stability.
> > > > > > > > > > > > пн, 26 нояб. 2018 г. в 17:48, aplatonov <
> > aplatonovv@gmail.com>:
> > > > > > > > > > > > >
> > > > > > > > > > > > > It should be noticed that additional parameter
> > TEST_SCALE_FACTOR was added.
> > > > > > > > > > > > > This parameter with ScaleFactorUtil methods can be
> > used for test size
> > > > > > > > > > > > > scaling for different runs (like ordinary and
> > nightly RunALLs). If someone
> > > > > > > > > > > > > want to distinguish these builds he/she can apply
> > scaling methods from
> > > > > > > > > > > > > ScaleFactorUtil in own tests. For nightly test
> > TEST_SCALE_FACTOR=1.0, for
> > > > > > > > > > > > > non-nightly builds TEST_SCALE_FACTOR<1.0. For
> > example in
> > > > > > > > > > > > > GridAbstractCacheInterceptorRebalanceTest test
> > ScaleFactorUtil was used for
> > > > > > > > > > > > > scaling count of iterations. I guess that
> > TEST_SCALE_FACTOR support will be
> > > > > > > > > > > > > added to runs at the same time with RunALL (nightly)
> > runs.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Sent from:
> > http://apache-ignite-developers.2346864.n4.nabble.com/
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Ivan Pavlukhin
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Best Regards, Vyacheslav D.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Best regards,
> > > > > > > > > > Ivan Pavlukhin
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best Regards, Vyacheslav D.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Ivan Pavlukhin
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best Regards, Vyacheslav D.
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> >
--
Best regards,
Ivan Pavlukhin
Re: Brainstorm: Make TC Run All faster
Posted by Павлухин Иван <vo...@gmail.com>.
Hi Dmitriy,
Thank you for adding staistics! With a good measure we can easily
compare different setups.
ср, 1 мая 2019 г. в 08:34, Павлухин Иван <vo...@gmail.com>:
>
> Nikolay,
>
> > I thought by "faster" we all meant time between One push RunAll button and one get results.
> > What else definition are possible here?
>
> But this "faster" depends on a current load. E.g. if we have a lot of
> idle slaves then incresead parallelism will allow a build to complete
> "faster". If we have many slaves busy than parallelism might "slow"
> down an execution. I predict that you are mostly interested in average
> metrics when TC has a build queue as it usually have on weekdays. If
> so then aggregatin other related jobs into "Build Apache Ignite" might
> speed execution of RunAll measured in wall clock time.
>
> And also I would like to return to a point I already mentioned earlier
> in other conversations. Our CI environment suites perfectly for an
> agile way of working. I suppose that we could try different setups
> almost freely and see if it gives a desired effect (and rollback if
> no). I am ok to go with Vyacheslav proposal. But I would like to
> observe some metrics showing that builds become running faster than
> before. Can we do this?
>
> вт, 30 апр. 2019 г. в 17:42, Dmitriy Pavlov <dp...@apache.org>:
> >
> > Hi, One more metric we can introduce is efficiency. Test run time/overall
> > time. The goal of launching RunAll is to obtain tests results.
> >
> > I've added more statistics to be displayed in the TC Bot, so now you may
> > click 'more' button near 'chain results' caption and you'll see sum of
> > statistics for all chain or individual suite. Latest RunAll gives the
> > following:
> > Overall time 57h 52m 41.188s (
> > - Build Net Time: 45h 10m 31.686s,
> > - Tests: 38h 58m 30.8s,
> > - Src. Update: 2h 29m 51.14s,
> > - Artifacts Publishing: 26m 38.295s,
> > - Dependencies Resolving: 9h 45m 1.809s,
> > - Timeouts: 3h 54m 16.071s)
> >
> > Currently, Efficiency is around 67-72%
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 29 апр. 2019 г. в 12:57, Nikolay Izhikov <ni...@apache.org>:
> >
> > > Ivan,
> > >
> > > I thought by "faster" we all meant time between One push RunAll button and
> > > one get results.
> > > What else definition are possible here?
> > >
> > > В Пн, 29/04/2019 в 12:04 +0300, Павлухин Иван пишет:
> > > > Nikolay,
> > > >
> > > > > Why we should imagine it?
> > > >
> > > > Only in order to understand what do we mean by "faster". My question was:
> > > >
> > > > 1st approach will take less agent time in sum. 2nd approach will
> > > > complete faster in wall clock time. And your main concern is "total
> > > > agent time". Am I right?
> > > >
> > > > пн, 29 апр. 2019 г. в 12:01, Nikolay Izhikov <ni...@apache.org>:
> > > > >
> > > > > > Let's imagine that we have an infinite number of agents
> > > > >
> > > > > Why we should imagine it?
> > > > > We don't have infinite number of agents.
> > > > >
> > > > > And we have several concurrent Run All.
> > > > >
> > > > >
> > > > > В Пн, 29/04/2019 в 11:50 +0300, Павлухин Иван пишет:
> > > > > > Vyacheslav,
> > > > > >
> > > > > > I finally figured out that "faster" means "total agent time".
> > > > > >
> > > > > > Let's imagine that we have an infinite number of agents. And 2
> > > approaches:
> > > > > > 1. Uber "Build Apache Ignite" containing all checks.
> > > > > > 2. Separate jobs for compilation, checkstyle and etc.
> > > > > >
> > > > > > 1st approach will take less agent time in sum. 2nd approach will
> > > > > > complete faster in wall clock time. And your main concern is "total
> > > > > > agent time". Am I right?
> > > > > >
> > > > > > пн, 29 апр. 2019 г. в 11:42, Vyacheslav Daradur <daradurvs@gmail.com
> > > >:
> > > > > > >
> > > > > > > Hi, Ivan,
> > > > > > >
> > > > > > > We are in the thread "Make TC Run All faster", so the main benefit
> > > is
> > > > > > > to make TC faster :)
> > > > > > >
> > > > > > > Advantages:
> > > > > > > - 1 TC agent required instead of 4;
> > > > > > > - RunAll will be faster, in case of builds queue;
> > > > > > >
> > > > > > > Also, the "licenses" profile is included in the step of a release
> > > > > > > build. I believe check-style also should be included.
> > > > > > >
> > > > > > > The generation of Javadocs is an optional step at preparing the
> > > > > > > release, but its check on TC takes significant time in case of the
> > > > > > > separate build.
> > > > > > >
> > > > > > > > > Returning to "Build Apache Ignite" it seems to me that
> > > ideally, it can
> > > > > > >
> > > > > > > be hierarchical.
> > > > > > >
> > > > > > > I agree, all the checks may be set as a separate step in the
> > > build's
> > > > > > > configuration. This helps with the main problem I'm trying to
> > > solve -
> > > > > > > resolving of dependencies which takes the most time of the builds.
> > > > > > >
> > > > > > > On Mon, Apr 29, 2019 at 11:24 AM Павлухин Иван <
> > > vololo100@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Vyacheslav, Maxim,
> > > > > > > >
> > > > > > > > Can we once again outline what benefits aggregated "Build Apache
> > > > > > > > Ignite" performing various checks has comparing to a modularized
> > > > > > > > approach in which separate builds perform separate tasks?
> > > > > > > >
> > > > > > > > For example, modularized approach looks nice because it is
> > > similar to
> > > > > > > > good practices in software development where we separate
> > > > > > > > responsibilities between different classes instead of
> > > aggregating them
> > > > > > > > into a single class. And as usual multiple classes works together
> > > > > > > > coordinating by a class from upper level. So, in fact it is a
> > > > > > > > hierarchical structure.
> > > > > > > >
> > > > > > > > Returning to "Build Apache Ignite" it seems to me that ideally
> > > it can
> > > > > > > > be hierarchical. There is a top level compilation (assembly?)
> > > job but
> > > > > > > > it is always clear what tasks does it perform (check style, check
> > > > > > > > license and other subjobs).
> > > > > > > >
> > > > > > > > пт, 26 апр. 2019 г. в 17:06, Maxim Muzafarov <maxmuzaf@gmail.com
> > > >:
> > > > > > > > >
> > > > > > > > > Folks,
> > > > > > > > >
> > > > > > > > > +1 for merging all these suites into the single one. All these
> > > suites
> > > > > > > > > (Build Apache Ignite, Javadoc, Licenses Header, Checkstyle)
> > > required
> > > > > > > > > to be `green` all the time. So, we can consider making them a
> > > part of
> > > > > > > > > build Apache Ignite procedure.
> > > > > > > > >
> > > > > > > > > Also, I'd suggest going deeper. We can try to merge `Licenses
> > > Header`
> > > > > > > > > into the `Code style checker` [1]. This will simplify the code
> > > > > > > > > checking process.
> > > > > > > > >
> > > > > > > > > [1] http://checkstyle.sourceforge.net/config_header.html
> > > > > > > > >
> > > > > > > > > On Fri, 26 Apr 2019 at 13:17, Vyacheslav Daradur <
> > > daradurvs@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Ivan, you are right, I meant to combine them into one.
> > > > > > > > > >
> > > > > > > > > > Here is a build [1], with enabled profiles (check-licenses,
> > > > > > > > > > checkstyle) and check of javadoc to show the idea.
> > > > > > > > > >
> > > > > > > > > > Seems it takes ~15 minutes.
> > > > > > > > > >
> > > > > > > > > > [1]
> > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ExperimentalBuildApacheIgniteJavadocLicensesHeaderCheckstyle&branch_IgniteTests24Java8=
> > > <default>;;
> > > > > > > > > >
> > > > > > > > > > On Fri, Apr 26, 2019 at 12:06 PM Павлухин Иван <
> > > vololo100@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Vyacheslav,
> > > > > > > > > > >
> > > > > > > > > > > What do you mean by uniting?
> > > > > > > > > > >
> > > > > > > > > > > For me it looks like [Javadocs] and [Check Code Style] are
> > > not so time
> > > > > > > > > > > consuming comparing to tests, are not they? Do you suggest
> > > to combine
> > > > > > > > > > > mentioned 4 jobs into one? How long will it run in a such
> > > case?
> > > > > > > > > > >
> > > > > > > > > > > чт, 25 апр. 2019 г. в 10:50, Vyacheslav Daradur <
> > > daradurvs@gmail.com>:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Igniters,
> > > > > > > > > > > >
> > > > > > > > > > > > At the moment we have several separated test suites:
> > > > > > > > > > > > * ~Build Apache Ignite~ _ ~10..20mins
> > > > > > > > > > > > * [Javadocs] _ ~10mins
> > > > > > > > > > > > * [Licenses Headers] _ ~1min
> > > > > > > > > > > > * [Check Code Style] _ ~7min
> > > > > > > > > > > > The most time of each build (except Licenses Headers) is
> > > taken by
> > > > > > > > > > > > dependency resolving.
> > > > > > > > > > > >
> > > > > > > > > > > > Their main goal is a check that the project is built
> > > properly.
> > > > > > > > > > > >
> > > > > > > > > > > > Also, profiles of [Javadocs], [Licenses Headers] uses at
> > > the step of
> > > > > > > > > > > > preparing release (see DEVNOTES.txt) that means they are
> > > important.
> > > > > > > > > > > >
> > > > > > > > > > > > I'd suggest uniting the builds, this should reduce the
> > > time of tests
> > > > > > > > > > > > on ~15 minutes and releases agents.
> > > > > > > > > > > >
> > > > > > > > > > > > What do you think?
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 27, 2018 at 3:56 PM Павлухин Иван <
> > > vololo100@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Roman,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Do you have some expectations how faster "correlated"
> > > tests
> > > > > > > > > > > > > elimination will make Run All? Also do you have a
> > > vision how can we
> > > > > > > > > > > > > determine such "correlated" tests, can we do it
> > > relatively fast?
> > > > > > > > > > > > >
> > > > > > > > > > > > > But all in all, I am not sure that reducing a group of
> > > correlated
> > > > > > > > > > > > > tests to only one test can show good stability.
> > > > > > > > > > > > > пн, 26 нояб. 2018 г. в 17:48, aplatonov <
> > > aplatonovv@gmail.com>:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It should be noticed that additional parameter
> > > TEST_SCALE_FACTOR was added.
> > > > > > > > > > > > > > This parameter with ScaleFactorUtil methods can be
> > > used for test size
> > > > > > > > > > > > > > scaling for different runs (like ordinary and
> > > nightly RunALLs). If someone
> > > > > > > > > > > > > > want to distinguish these builds he/she can apply
> > > scaling methods from
> > > > > > > > > > > > > > ScaleFactorUtil in own tests. For nightly test
> > > TEST_SCALE_FACTOR=1.0, for
> > > > > > > > > > > > > > non-nightly builds TEST_SCALE_FACTOR<1.0. For
> > > example in
> > > > > > > > > > > > > > GridAbstractCacheInterceptorRebalanceTest test
> > > ScaleFactorUtil was used for
> > > > > > > > > > > > > > scaling count of iterations. I guess that
> > > TEST_SCALE_FACTOR support will be
> > > > > > > > > > > > > > added to runs at the same time with RunALL (nightly)
> > > runs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Sent from:
> > > http://apache-ignite-developers.2346864.n4.nabble.com/
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Ivan Pavlukhin
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Best Regards, Vyacheslav D.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Ivan Pavlukhin
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Best Regards, Vyacheslav D.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best regards,
> > > > > > > > Ivan Pavlukhin
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Regards, Vyacheslav D.
> > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
--
Best regards,
Ivan Pavlukhin