You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Dmitrii Ryabov <so...@gmail.com> on 2018/09/03 15:20:17 UTC
Re: Workflow improvement
Hi, Dmitriy,
I created a table with possible blockers [1] for the page with the latest
build analysis. Anton K. has reviewed it. Can you check and merge it?
Also, I created a button to notify PR about analisis results [2]. I used
GitHub statuses (example [3]), but to work it needs a token with push
permission. As Apache PMC, can you acquire such token? I think statuses are
much more notable than comments.
[1] https://issues.apache.org/jira/browse/IGNITE-9376
[2] https://issues.apache.org/jira/browse/IGNITE-9377
[3] https://github.com/SomeFire/ignite-teamcity-bot/pulls
2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> Not sure I understood what bot statistic reduction means.
>
> As far as I understood, you also mean locating failure automatically with
> re-runs on specific git revisions (hashes). I found it will be a good
> contribution to TC Bot for both flaky and non-flaky failures detection. E.g
> failure occurred after 10 commits merged to master. How to find out bad
> commit? Automatic location (analog of bisecting, but using TC test) will be
> really helpful. But it is not a simple contribution.
>
> пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <so...@gmail.com>:
>
> > Hi, Dmitriy,
> >
> > Good, I'll create tickets for this steps.
> >
> > Stable suites are a good idea, but also we need to do some actions when a
> > flaky test will appear in stable suite of master branch run. To find out
> a
> > guilty commit, we should run previous commits on empty TeamCity's queue.
> > This runs will reduce bot's statistics for the latest commits.
> >
> > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> >
> > > Hi Dmitriy,
> > >
> > > Sounds like a plan ;) I totally agree.
> > > Just one proposal: I would like to avoid hiding any test failures. 2
> > > separate tables named 'Possible Blockers' and 'Other failures' should
> be
> > > completely clear. Comment to PR can contain only first category.
> > >
> > > We had a private discussion with Anton K. and he proposed a very
> > > interesting thing, I would like to bring it here. We can add
> > configuration
> > > into TC bot and mark some tests and some suites as 'Known Stable' It
> > means
> > > that suite or test failure should be considered as a blocker for merge
> > > every time it fails, even if fail rate is non-zero. Very first list of
> > such
> > > suites are
> > > - Build Apache Ignite
> > > - License check
> > > And tests:
> > > - .Net API Parity check
> > > Please share your vision.
> > >
> > > Meanwhile, I've updated the TC bot.
> > > - Underlying Apache Ignite DB was refactored to use lower partitions
> > count,
> > > so restart should be faster.
> > > - Now 100 builds are saved as our failure rate statistics basis.
> > Flakiness
> > > border was not changed, so more test now will be considered as flaky.
> > > - Now 4 builds required to be failed or timed out in a row before
> > > notification is sent.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <so...@gmail.com>:
> > >
> > > > Hi, Dmitriy,
> > > >
> > > > I propose the next steps:
> > > >
> > > > 1. Show current 0% tests in a separate table at the top of the
> analysis
> > > > results page. Thus, we'll see most suspicious (new or very flaky)
> tests
> > > > firstly. Or we can hide other tests under "More >>" button, like top
> > long
> > > > running tests.
> > > > 2. Create a button by clicking on which the info about 0% tests will
> be
> > > > written in the PR.
> > > > 3. Replace button by webhook for TeamCity (for Run All), which will
> > start
> > > > analysis on TCH and write results in the PR.
> > > > 4. ...
> > > >
> > > > What do you think?
> > > >
> > > >
> > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <so...@gmail.com>:
> > > >
> > > > > I think we should check not N last runs, but all runs in last N
> days.
> > > > >
> > > > > A simple rule to detect flaky fails by hands - get test history
> > ordered
> > > > > by TEST_STATUS_DESC and check its date. As I see, we can get list
> of
> > > > > failures from TC. We don't need to check successfull runs, because
> > they
> > > > are
> > > > > uninformative for our needs.
> > > > >
> > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> > > > >
> > > > >> Hi Dmitriy,
> > > > >>
> > > > >> The Bot is able to detect a frequent change of test status, but
> > > > currently
> > > > >> only 50 last runs count. Same is true for the failure rate.
> > > > >>
> > > > >> This value can be easily changed to 70 or 100, moreover, the auto
> > > > >> trigger feature gives us much more builds.
> > > > >>
> > > > >> We can improve these rules. We can add not only status change, but
> > > > status
> > > > >> change without any code changes. We can somehow save this data in
> > > > RunStat
> > > > >> class. Let's create a better rule, and later we can code it.
> > > > >>
> > > > >> Sincerely,
> > > > >> Dmitriy Pavlov
> > > > >>
> > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <
> somefireone@gmail.com
> > >:
> > > > >>
> > > > >> > I think plugin will be more pretty looking, but comments can
> > contain
> > > > any
> > > > >> > information, so they can be more usefull. I agree with your idea
> > to
> > > > >> create
> > > > >> > bot instead of plugin.
> > > > >> >
> > > > >> > As for fail rate - I'm not sure it is working as you describe.
> > > > >> > I'm looking on my runAll [1]. There is
> > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated`
> > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and
> > fails
> > > in
> > > > >> > master branch [2].
> > > > >> >
> > > > >> > [1]
> > > > >> >
> > > > >> > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=I
> > > > >>
> > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest
> > > > >> > [2]
> > > > >> >
> > > > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTe
> > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470
> > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests
> > > > >> 24Java8=__all_branches__&itemsCount=50
> > > > >> >
> > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <
> dpavlov.spb@gmail.com
> > >:
> > > > >> >
> > > > >> > > Hi Dmitrii,
> > > > >> > >
> > > > >> > > I'm not sure we're able to install Github apps to Apache
> > mirrors.
> > > > >> > >
> > > > >> > > The simplest solution, what can be as efficient as a plugin,
> is
> > > fake
> > > > >> > MTCGA
> > > > >> > > bot account in Github, which will provide PR comments using
> > Github
> > > > >> > program
> > > > >> > > interface. What do you think?
> > > > >> > >
> > > > >> > > A new test failure can be identified by the Ignite TC Bot by
> > > master
> > > > >> > recent
> > > > >> > > fail rate = 0.0%. The same rule can be applied to timed out
> > > suites.
> > > > >> > >
> > > > >> > > Sincerely,
> > > > >> > > Dmitriy Pavlov
> > > > >> > >
> > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <
> > > somefireone@gmail.com
> > > > >:
> > > > >> > >
> > > > >> > > > Hello, Igniters!
> > > > >> > > >
> > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – we
> > need
> > > an
> > > > >> easy
> > > > >> > > way
> > > > >> > > > to get list of failed tests that don’t fall in the master
> > > branch.
> > > > >> These
> > > > >> > > > tests should:
> > > > >> > > > * fail in the PR
> > > > >> > > > * not fail in the master
> > > > >> > > > * not be flaky.
> > > > >> > > >
> > > > >> > > > Also, I want to suggest to create a GitHub plugin, which
> will
> > > > >> notify PR
> > > > >> > > if
> > > > >> > > > it has such tests. PR will have a marker, which
> > allows/prohibits
> > > > >> merge.
> > > > >> > > > This marker will be shown near PR conflicts.
> > > > >> > > >
> > > > >> > > > Allowing marker will be shown in case:
> > > > >> > > > * no new fails.
> > > > >> > > >
> > > > >> > > > Prohibiting marker will be shown in cases:
> > > > >> > > > * new fails – tests must be fixed.
> > > > >> > > > * new timed out test suite – suite should be restarted or
> > tests
> > > > >> must be
> > > > >> > > > fixed.
> > > > >> > > > * runAll wasn’t launched – tests must be launched.
> > > > >> > > >
> > > > >> > > > This will make test checks much faster and easier. Also,
> this
> > > will
> > > > >> > > decrease
> > > > >> > > > the number of merges with new failed tests made by
> inattention
> > > to
> > > > >> the
> > > > >> > > > tests.
> > > > >> > > >
> > > > >> > > > Further, we can expand the plugin by adding new checks,
> > showing
> > > PR
> > > > >> > > quality.
> > > > >> > > >
> > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Workflow improvement
Posted by Dmitrii Ryabov <so...@gmail.com>.
Yep, it's working. Thank you.
2018-09-13 19:09 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> Hi Dmitriy,
>
> Sure, I agree with extra permissions assignment. Done.
>
> Could you please check if you can manage your builds now?
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 13 сент. 2018 г. в 16:05, Dmitrii Ryabov <so...@gmail.com>:
>
> > Hi, Dmitriy,
> >
> > Can you give me rights to stop my builds on TeamCity? Working on the
> TCH, I
> > run a lot of builds, and I don't like to ask other people to stop builds
> > too often.
> >
> > 2018-09-10 23:53 GMT+03:00 Dmitrii Ryabov <so...@gmail.com>:
> >
> > > Hi, Dmitriy,
> > >
> > > I made PRs in my fork for test purposes. Real PRs were made to the
> GitHub
> > > mirror, and one of them is already merged by D. Govorukhin. PR with
> > GitHub
> > > statuses [1] is ready for review. PR with JIRA comment will be ready
> in a
> > > few days.
> > >
> > > [1] https://github.com/apache/ignite-teamcity-bot/pull/5
> > >
> > > 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> > >
> > >> Hi Dmitrii,
> > >>
> > >> I deployed change with blockers summary of failures at the top of PR
> > >> result
> > >> page. The Bot is migrating entries now, I hope it will be done in 1-4
> > >> hours.
> > >>
> > >> I noticed your PRs are created in your own fork, not in
> > >> https://github.com/apache/ignite-teamcity-bot
> > >> Could you please create unmerged PR in GitHub mirror so it could be
> > merged
> > >> after reviewing by the apply-pr script.
> > >>
> > >> Sincerely,
> > >> Dmitriy Pavlov
> > >>
> > >> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <so...@gmail.com>:
> > >>
> > >> > Hi, Dmitriy,
> > >> >
> > >> > I created a table with possible blockers [1] for the page with the
> > >> latest
> > >> > build analysis. Anton K. has reviewed it. Can you check and merge
> it?
> > >> >
> > >> > Also, I created a button to notify PR about analisis results [2]. I
> > used
> > >> > GitHub statuses (example [3]), but to work it needs a token with
> push
> > >> > permission. As Apache PMC, can you acquire such token? I think
> > statuses
> > >> are
> > >> > much more notable than comments.
> > >> >
> > >> > [1] https://issues.apache.org/jira/browse/IGNITE-9376
> > >> > [2] https://issues.apache.org/jira/browse/IGNITE-9377
> > >> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls
> > >> >
> > >> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> > >> >
> > >> > > Not sure I understood what bot statistic reduction means.
> > >> > >
> > >> > > As far as I understood, you also mean locating failure
> automatically
> > >> with
> > >> > > re-runs on specific git revisions (hashes). I found it will be a
> > good
> > >> > > contribution to TC Bot for both flaky and non-flaky failures
> > >> detection.
> > >> > E.g
> > >> > > failure occurred after 10 commits merged to master. How to find
> out
> > >> bad
> > >> > > commit? Automatic location (analog of bisecting, but using TC
> test)
> > >> will
> > >> > be
> > >> > > really helpful. But it is not a simple contribution.
> > >> > >
> > >> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <
> somefireone@gmail.com
> > >:
> > >> > >
> > >> > > > Hi, Dmitriy,
> > >> > > >
> > >> > > > Good, I'll create tickets for this steps.
> > >> > > >
> > >> > > > Stable suites are a good idea, but also we need to do some
> actions
> > >> > when a
> > >> > > > flaky test will appear in stable suite of master branch run. To
> > find
> > >> > out
> > >> > > a
> > >> > > > guilty commit, we should run previous commits on empty
> TeamCity's
> > >> > queue.
> > >> > > > This runs will reduce bot's statistics for the latest commits.
> > >> > > >
> > >> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <
> dpavlov.spb@gmail.com
> > >:
> > >> > > >
> > >> > > > > Hi Dmitriy,
> > >> > > > >
> > >> > > > > Sounds like a plan ;) I totally agree.
> > >> > > > > Just one proposal: I would like to avoid hiding any test
> > >> failures. 2
> > >> > > > > separate tables named 'Possible Blockers' and 'Other failures'
> > >> should
> > >> > > be
> > >> > > > > completely clear. Comment to PR can contain only first
> category.
> > >> > > > >
> > >> > > > > We had a private discussion with Anton K. and he proposed a
> very
> > >> > > > > interesting thing, I would like to bring it here. We can add
> > >> > > > configuration
> > >> > > > > into TC bot and mark some tests and some suites as 'Known
> > Stable'
> > >> It
> > >> > > > means
> > >> > > > > that suite or test failure should be considered as a blocker
> for
> > >> > merge
> > >> > > > > every time it fails, even if fail rate is non-zero. Very first
> > >> list
> > >> > of
> > >> > > > such
> > >> > > > > suites are
> > >> > > > > - Build Apache Ignite
> > >> > > > > - License check
> > >> > > > > And tests:
> > >> > > > > - .Net API Parity check
> > >> > > > > Please share your vision.
> > >> > > > >
> > >> > > > > Meanwhile, I've updated the TC bot.
> > >> > > > > - Underlying Apache Ignite DB was refactored to use lower
> > >> partitions
> > >> > > > count,
> > >> > > > > so restart should be faster.
> > >> > > > > - Now 100 builds are saved as our failure rate statistics
> basis.
> > >> > > > Flakiness
> > >> > > > > border was not changed, so more test now will be considered as
> > >> flaky.
> > >> > > > > - Now 4 builds required to be failed or timed out in a row
> > before
> > >> > > > > notification is sent.
> > >> > > > >
> > >> > > > > Sincerely,
> > >> > > > > Dmitriy Pavlov
> > >> > > > >
> > >> > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <
> > >> somefireone@gmail.com>:
> > >> > > > >
> > >> > > > > > Hi, Dmitriy,
> > >> > > > > >
> > >> > > > > > I propose the next steps:
> > >> > > > > >
> > >> > > > > > 1. Show current 0% tests in a separate table at the top of
> the
> > >> > > analysis
> > >> > > > > > results page. Thus, we'll see most suspicious (new or very
> > >> flaky)
> > >> > > tests
> > >> > > > > > firstly. Or we can hide other tests under "More >>" button,
> > like
> > >> > top
> > >> > > > long
> > >> > > > > > running tests.
> > >> > > > > > 2. Create a button by clicking on which the info about 0%
> > tests
> > >> > will
> > >> > > be
> > >> > > > > > written in the PR.
> > >> > > > > > 3. Replace button by webhook for TeamCity (for Run All),
> which
> > >> will
> > >> > > > start
> > >> > > > > > analysis on TCH and write results in the PR.
> > >> > > > > > 4. ...
> > >> > > > > >
> > >> > > > > > What do you think?
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <
> > somefireone@gmail.com
> > >> >:
> > >> > > > > >
> > >> > > > > > > I think we should check not N last runs, but all runs in
> > last
> > >> N
> > >> > > days.
> > >> > > > > > >
> > >> > > > > > > A simple rule to detect flaky fails by hands - get test
> > >> history
> > >> > > > ordered
> > >> > > > > > > by TEST_STATUS_DESC and check its date. As I see, we can
> get
> > >> list
> > >> > > of
> > >> > > > > > > failures from TC. We don't need to check successfull runs,
> > >> > because
> > >> > > > they
> > >> > > > > > are
> > >> > > > > > > uninformative for our needs.
> > >> > > > > > >
> > >> > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <
> > >> dpavlov.spb@gmail.com
> > >> > >:
> > >> > > > > > >
> > >> > > > > > >> Hi Dmitriy,
> > >> > > > > > >>
> > >> > > > > > >> The Bot is able to detect a frequent change of test
> status,
> > >> but
> > >> > > > > > currently
> > >> > > > > > >> only 50 last runs count. Same is true for the failure
> rate.
> > >> > > > > > >>
> > >> > > > > > >> This value can be easily changed to 70 or 100, moreover,
> > the
> > >> > auto
> > >> > > > > > >> trigger feature gives us much more builds.
> > >> > > > > > >>
> > >> > > > > > >> We can improve these rules. We can add not only status
> > >> change,
> > >> > but
> > >> > > > > > status
> > >> > > > > > >> change without any code changes. We can somehow save this
> > >> data
> > >> > in
> > >> > > > > > RunStat
> > >> > > > > > >> class. Let's create a better rule, and later we can code
> > it.
> > >> > > > > > >>
> > >> > > > > > >> Sincerely,
> > >> > > > > > >> Dmitriy Pavlov
> > >> > > > > > >>
> > >> > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <
> > >> > > somefireone@gmail.com
> > >> > > > >:
> > >> > > > > > >>
> > >> > > > > > >> > I think plugin will be more pretty looking, but
> comments
> > >> can
> > >> > > > contain
> > >> > > > > > any
> > >> > > > > > >> > information, so they can be more usefull. I agree with
> > your
> > >> > idea
> > >> > > > to
> > >> > > > > > >> create
> > >> > > > > > >> > bot instead of plugin.
> > >> > > > > > >> >
> > >> > > > > > >> > As for fail rate - I'm not sure it is working as you
> > >> describe.
> > >> > > > > > >> > I'm looking on my runAll [1]. There is
> > >> > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated`
> > >> > > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is
> flaky
> > >> and
> > >> > > > fails
> > >> > > > > in
> > >> > > > > > >> > master branch [2].
> > >> > > > > > >> >
> > >> > > > > > >> > [1]
> > >> > > > > > >> >
> > >> > > > > > >> > https://mtcga.gridgain.com/pr.
> > >> html?serverId=apache&suiteId=I
> > >> > > > > > >>
> > >> > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&
> > >> action=Latest
> > >> > > > > > >> > [2]
> > >> > > > > > >> >
> > >> > > > > > >> > https://ci.ignite.apache.org/p
> > >> roject.html?projectId=IgniteTe
> > >> > > > > > >> sts24Java8&buildTypeId=&tab=
> testDetails&testNameId=5628470
> > >> > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests
> > >> > > > > > >> 24Java8=__all_branches__&itemsCount=50
> > >> > > > > > >> >
> > >> > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <
> > >> > > dpavlov.spb@gmail.com
> > >> > > > >:
> > >> > > > > > >> >
> > >> > > > > > >> > > Hi Dmitrii,
> > >> > > > > > >> > >
> > >> > > > > > >> > > I'm not sure we're able to install Github apps to
> > Apache
> > >> > > > mirrors.
> > >> > > > > > >> > >
> > >> > > > > > >> > > The simplest solution, what can be as efficient as a
> > >> plugin,
> > >> > > is
> > >> > > > > fake
> > >> > > > > > >> > MTCGA
> > >> > > > > > >> > > bot account in Github, which will provide PR comments
> > >> using
> > >> > > > Github
> > >> > > > > > >> > program
> > >> > > > > > >> > > interface. What do you think?
> > >> > > > > > >> > >
> > >> > > > > > >> > > A new test failure can be identified by the Ignite TC
> > >> Bot by
> > >> > > > > master
> > >> > > > > > >> > recent
> > >> > > > > > >> > > fail rate = 0.0%. The same rule can be applied to
> timed
> > >> out
> > >> > > > > suites.
> > >> > > > > > >> > >
> > >> > > > > > >> > > Sincerely,
> > >> > > > > > >> > > Dmitriy Pavlov
> > >> > > > > > >> > >
> > >> > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <
> > >> > > > > somefireone@gmail.com
> > >> > > > > > >:
> > >> > > > > > >> > >
> > >> > > > > > >> > > > Hello, Igniters!
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > I want to suggest improvement for TeamCity Helper
> [1]
> > >> – we
> > >> > > > need
> > >> > > > > an
> > >> > > > > > >> easy
> > >> > > > > > >> > > way
> > >> > > > > > >> > > > to get list of failed tests that don’t fall in the
> > >> master
> > >> > > > > branch.
> > >> > > > > > >> These
> > >> > > > > > >> > > > tests should:
> > >> > > > > > >> > > > * fail in the PR
> > >> > > > > > >> > > > * not fail in the master
> > >> > > > > > >> > > > * not be flaky.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > Also, I want to suggest to create a GitHub plugin,
> > >> which
> > >> > > will
> > >> > > > > > >> notify PR
> > >> > > > > > >> > > if
> > >> > > > > > >> > > > it has such tests. PR will have a marker, which
> > >> > > > allows/prohibits
> > >> > > > > > >> merge.
> > >> > > > > > >> > > > This marker will be shown near PR conflicts.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > Allowing marker will be shown in case:
> > >> > > > > > >> > > > * no new fails.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > Prohibiting marker will be shown in cases:
> > >> > > > > > >> > > > * new fails – tests must be fixed.
> > >> > > > > > >> > > > * new timed out test suite – suite should be
> > restarted
> > >> or
> > >> > > > tests
> > >> > > > > > >> must be
> > >> > > > > > >> > > > fixed.
> > >> > > > > > >> > > > * runAll wasn’t launched – tests must be launched.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > This will make test checks much faster and easier.
> > >> Also,
> > >> > > this
> > >> > > > > will
> > >> > > > > > >> > > decrease
> > >> > > > > > >> > > > the number of merges with new failed tests made by
> > >> > > inattention
> > >> > > > > to
> > >> > > > > > >> the
> > >> > > > > > >> > > > tests.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > Further, we can expand the plugin by adding new
> > checks,
> > >> > > > showing
> > >> > > > > PR
> > >> > > > > > >> > > quality.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot
> > >> > > > > > >> > > >
> > >> > > > > > >> > >
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Workflow improvement
Posted by Dmitriy Pavlov <dp...@gmail.com>.
Hi Dmitriy,
Sure, I agree with extra permissions assignment. Done.
Could you please check if you can manage your builds now?
Sincerely,
Dmitriy Pavlov
чт, 13 сент. 2018 г. в 16:05, Dmitrii Ryabov <so...@gmail.com>:
> Hi, Dmitriy,
>
> Can you give me rights to stop my builds on TeamCity? Working on the TCH, I
> run a lot of builds, and I don't like to ask other people to stop builds
> too often.
>
> 2018-09-10 23:53 GMT+03:00 Dmitrii Ryabov <so...@gmail.com>:
>
> > Hi, Dmitriy,
> >
> > I made PRs in my fork for test purposes. Real PRs were made to the GitHub
> > mirror, and one of them is already merged by D. Govorukhin. PR with
> GitHub
> > statuses [1] is ready for review. PR with JIRA comment will be ready in a
> > few days.
> >
> > [1] https://github.com/apache/ignite-teamcity-bot/pull/5
> >
> > 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> >
> >> Hi Dmitrii,
> >>
> >> I deployed change with blockers summary of failures at the top of PR
> >> result
> >> page. The Bot is migrating entries now, I hope it will be done in 1-4
> >> hours.
> >>
> >> I noticed your PRs are created in your own fork, not in
> >> https://github.com/apache/ignite-teamcity-bot
> >> Could you please create unmerged PR in GitHub mirror so it could be
> merged
> >> after reviewing by the apply-pr script.
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <so...@gmail.com>:
> >>
> >> > Hi, Dmitriy,
> >> >
> >> > I created a table with possible blockers [1] for the page with the
> >> latest
> >> > build analysis. Anton K. has reviewed it. Can you check and merge it?
> >> >
> >> > Also, I created a button to notify PR about analisis results [2]. I
> used
> >> > GitHub statuses (example [3]), but to work it needs a token with push
> >> > permission. As Apache PMC, can you acquire such token? I think
> statuses
> >> are
> >> > much more notable than comments.
> >> >
> >> > [1] https://issues.apache.org/jira/browse/IGNITE-9376
> >> > [2] https://issues.apache.org/jira/browse/IGNITE-9377
> >> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls
> >> >
> >> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> >> >
> >> > > Not sure I understood what bot statistic reduction means.
> >> > >
> >> > > As far as I understood, you also mean locating failure automatically
> >> with
> >> > > re-runs on specific git revisions (hashes). I found it will be a
> good
> >> > > contribution to TC Bot for both flaky and non-flaky failures
> >> detection.
> >> > E.g
> >> > > failure occurred after 10 commits merged to master. How to find out
> >> bad
> >> > > commit? Automatic location (analog of bisecting, but using TC test)
> >> will
> >> > be
> >> > > really helpful. But it is not a simple contribution.
> >> > >
> >> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <somefireone@gmail.com
> >:
> >> > >
> >> > > > Hi, Dmitriy,
> >> > > >
> >> > > > Good, I'll create tickets for this steps.
> >> > > >
> >> > > > Stable suites are a good idea, but also we need to do some actions
> >> > when a
> >> > > > flaky test will appear in stable suite of master branch run. To
> find
> >> > out
> >> > > a
> >> > > > guilty commit, we should run previous commits on empty TeamCity's
> >> > queue.
> >> > > > This runs will reduce bot's statistics for the latest commits.
> >> > > >
> >> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <dpavlov.spb@gmail.com
> >:
> >> > > >
> >> > > > > Hi Dmitriy,
> >> > > > >
> >> > > > > Sounds like a plan ;) I totally agree.
> >> > > > > Just one proposal: I would like to avoid hiding any test
> >> failures. 2
> >> > > > > separate tables named 'Possible Blockers' and 'Other failures'
> >> should
> >> > > be
> >> > > > > completely clear. Comment to PR can contain only first category.
> >> > > > >
> >> > > > > We had a private discussion with Anton K. and he proposed a very
> >> > > > > interesting thing, I would like to bring it here. We can add
> >> > > > configuration
> >> > > > > into TC bot and mark some tests and some suites as 'Known
> Stable'
> >> It
> >> > > > means
> >> > > > > that suite or test failure should be considered as a blocker for
> >> > merge
> >> > > > > every time it fails, even if fail rate is non-zero. Very first
> >> list
> >> > of
> >> > > > such
> >> > > > > suites are
> >> > > > > - Build Apache Ignite
> >> > > > > - License check
> >> > > > > And tests:
> >> > > > > - .Net API Parity check
> >> > > > > Please share your vision.
> >> > > > >
> >> > > > > Meanwhile, I've updated the TC bot.
> >> > > > > - Underlying Apache Ignite DB was refactored to use lower
> >> partitions
> >> > > > count,
> >> > > > > so restart should be faster.
> >> > > > > - Now 100 builds are saved as our failure rate statistics basis.
> >> > > > Flakiness
> >> > > > > border was not changed, so more test now will be considered as
> >> flaky.
> >> > > > > - Now 4 builds required to be failed or timed out in a row
> before
> >> > > > > notification is sent.
> >> > > > >
> >> > > > > Sincerely,
> >> > > > > Dmitriy Pavlov
> >> > > > >
> >> > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <
> >> somefireone@gmail.com>:
> >> > > > >
> >> > > > > > Hi, Dmitriy,
> >> > > > > >
> >> > > > > > I propose the next steps:
> >> > > > > >
> >> > > > > > 1. Show current 0% tests in a separate table at the top of the
> >> > > analysis
> >> > > > > > results page. Thus, we'll see most suspicious (new or very
> >> flaky)
> >> > > tests
> >> > > > > > firstly. Or we can hide other tests under "More >>" button,
> like
> >> > top
> >> > > > long
> >> > > > > > running tests.
> >> > > > > > 2. Create a button by clicking on which the info about 0%
> tests
> >> > will
> >> > > be
> >> > > > > > written in the PR.
> >> > > > > > 3. Replace button by webhook for TeamCity (for Run All), which
> >> will
> >> > > > start
> >> > > > > > analysis on TCH and write results in the PR.
> >> > > > > > 4. ...
> >> > > > > >
> >> > > > > > What do you think?
> >> > > > > >
> >> > > > > >
> >> > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <
> somefireone@gmail.com
> >> >:
> >> > > > > >
> >> > > > > > > I think we should check not N last runs, but all runs in
> last
> >> N
> >> > > days.
> >> > > > > > >
> >> > > > > > > A simple rule to detect flaky fails by hands - get test
> >> history
> >> > > > ordered
> >> > > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get
> >> list
> >> > > of
> >> > > > > > > failures from TC. We don't need to check successfull runs,
> >> > because
> >> > > > they
> >> > > > > > are
> >> > > > > > > uninformative for our needs.
> >> > > > > > >
> >> > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <
> >> dpavlov.spb@gmail.com
> >> > >:
> >> > > > > > >
> >> > > > > > >> Hi Dmitriy,
> >> > > > > > >>
> >> > > > > > >> The Bot is able to detect a frequent change of test status,
> >> but
> >> > > > > > currently
> >> > > > > > >> only 50 last runs count. Same is true for the failure rate.
> >> > > > > > >>
> >> > > > > > >> This value can be easily changed to 70 or 100, moreover,
> the
> >> > auto
> >> > > > > > >> trigger feature gives us much more builds.
> >> > > > > > >>
> >> > > > > > >> We can improve these rules. We can add not only status
> >> change,
> >> > but
> >> > > > > > status
> >> > > > > > >> change without any code changes. We can somehow save this
> >> data
> >> > in
> >> > > > > > RunStat
> >> > > > > > >> class. Let's create a better rule, and later we can code
> it.
> >> > > > > > >>
> >> > > > > > >> Sincerely,
> >> > > > > > >> Dmitriy Pavlov
> >> > > > > > >>
> >> > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <
> >> > > somefireone@gmail.com
> >> > > > >:
> >> > > > > > >>
> >> > > > > > >> > I think plugin will be more pretty looking, but comments
> >> can
> >> > > > contain
> >> > > > > > any
> >> > > > > > >> > information, so they can be more usefull. I agree with
> your
> >> > idea
> >> > > > to
> >> > > > > > >> create
> >> > > > > > >> > bot instead of plugin.
> >> > > > > > >> >
> >> > > > > > >> > As for fail rate - I'm not sure it is working as you
> >> describe.
> >> > > > > > >> > I'm looking on my runAll [1]. There is
> >> > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated`
> >> > > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky
> >> and
> >> > > > fails
> >> > > > > in
> >> > > > > > >> > master branch [2].
> >> > > > > > >> >
> >> > > > > > >> > [1]
> >> > > > > > >> >
> >> > > > > > >> > https://mtcga.gridgain.com/pr.
> >> html?serverId=apache&suiteId=I
> >> > > > > > >>
> >> > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&
> >> action=Latest
> >> > > > > > >> > [2]
> >> > > > > > >> >
> >> > > > > > >> > https://ci.ignite.apache.org/p
> >> roject.html?projectId=IgniteTe
> >> > > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470
> >> > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests
> >> > > > > > >> 24Java8=__all_branches__&itemsCount=50
> >> > > > > > >> >
> >> > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <
> >> > > dpavlov.spb@gmail.com
> >> > > > >:
> >> > > > > > >> >
> >> > > > > > >> > > Hi Dmitrii,
> >> > > > > > >> > >
> >> > > > > > >> > > I'm not sure we're able to install Github apps to
> Apache
> >> > > > mirrors.
> >> > > > > > >> > >
> >> > > > > > >> > > The simplest solution, what can be as efficient as a
> >> plugin,
> >> > > is
> >> > > > > fake
> >> > > > > > >> > MTCGA
> >> > > > > > >> > > bot account in Github, which will provide PR comments
> >> using
> >> > > > Github
> >> > > > > > >> > program
> >> > > > > > >> > > interface. What do you think?
> >> > > > > > >> > >
> >> > > > > > >> > > A new test failure can be identified by the Ignite TC
> >> Bot by
> >> > > > > master
> >> > > > > > >> > recent
> >> > > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed
> >> out
> >> > > > > suites.
> >> > > > > > >> > >
> >> > > > > > >> > > Sincerely,
> >> > > > > > >> > > Dmitriy Pavlov
> >> > > > > > >> > >
> >> > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <
> >> > > > > somefireone@gmail.com
> >> > > > > > >:
> >> > > > > > >> > >
> >> > > > > > >> > > > Hello, Igniters!
> >> > > > > > >> > > >
> >> > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1]
> >> – we
> >> > > > need
> >> > > > > an
> >> > > > > > >> easy
> >> > > > > > >> > > way
> >> > > > > > >> > > > to get list of failed tests that don’t fall in the
> >> master
> >> > > > > branch.
> >> > > > > > >> These
> >> > > > > > >> > > > tests should:
> >> > > > > > >> > > > * fail in the PR
> >> > > > > > >> > > > * not fail in the master
> >> > > > > > >> > > > * not be flaky.
> >> > > > > > >> > > >
> >> > > > > > >> > > > Also, I want to suggest to create a GitHub plugin,
> >> which
> >> > > will
> >> > > > > > >> notify PR
> >> > > > > > >> > > if
> >> > > > > > >> > > > it has such tests. PR will have a marker, which
> >> > > > allows/prohibits
> >> > > > > > >> merge.
> >> > > > > > >> > > > This marker will be shown near PR conflicts.
> >> > > > > > >> > > >
> >> > > > > > >> > > > Allowing marker will be shown in case:
> >> > > > > > >> > > > * no new fails.
> >> > > > > > >> > > >
> >> > > > > > >> > > > Prohibiting marker will be shown in cases:
> >> > > > > > >> > > > * new fails – tests must be fixed.
> >> > > > > > >> > > > * new timed out test suite – suite should be
> restarted
> >> or
> >> > > > tests
> >> > > > > > >> must be
> >> > > > > > >> > > > fixed.
> >> > > > > > >> > > > * runAll wasn’t launched – tests must be launched.
> >> > > > > > >> > > >
> >> > > > > > >> > > > This will make test checks much faster and easier.
> >> Also,
> >> > > this
> >> > > > > will
> >> > > > > > >> > > decrease
> >> > > > > > >> > > > the number of merges with new failed tests made by
> >> > > inattention
> >> > > > > to
> >> > > > > > >> the
> >> > > > > > >> > > > tests.
> >> > > > > > >> > > >
> >> > > > > > >> > > > Further, we can expand the plugin by adding new
> checks,
> >> > > > showing
> >> > > > > PR
> >> > > > > > >> > > quality.
> >> > > > > > >> > > >
> >> > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot
> >> > > > > > >> > > >
> >> > > > > > >> > >
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Workflow improvement
Posted by Dmitrii Ryabov <so...@gmail.com>.
Hi, Dmitriy,
Can you give me rights to stop my builds on TeamCity? Working on the TCH, I
run a lot of builds, and I don't like to ask other people to stop builds
too often.
2018-09-10 23:53 GMT+03:00 Dmitrii Ryabov <so...@gmail.com>:
> Hi, Dmitriy,
>
> I made PRs in my fork for test purposes. Real PRs were made to the GitHub
> mirror, and one of them is already merged by D. Govorukhin. PR with GitHub
> statuses [1] is ready for review. PR with JIRA comment will be ready in a
> few days.
>
> [1] https://github.com/apache/ignite-teamcity-bot/pull/5
>
> 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
>
>> Hi Dmitrii,
>>
>> I deployed change with blockers summary of failures at the top of PR
>> result
>> page. The Bot is migrating entries now, I hope it will be done in 1-4
>> hours.
>>
>> I noticed your PRs are created in your own fork, not in
>> https://github.com/apache/ignite-teamcity-bot
>> Could you please create unmerged PR in GitHub mirror so it could be merged
>> after reviewing by the apply-pr script.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <so...@gmail.com>:
>>
>> > Hi, Dmitriy,
>> >
>> > I created a table with possible blockers [1] for the page with the
>> latest
>> > build analysis. Anton K. has reviewed it. Can you check and merge it?
>> >
>> > Also, I created a button to notify PR about analisis results [2]. I used
>> > GitHub statuses (example [3]), but to work it needs a token with push
>> > permission. As Apache PMC, can you acquire such token? I think statuses
>> are
>> > much more notable than comments.
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-9376
>> > [2] https://issues.apache.org/jira/browse/IGNITE-9377
>> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls
>> >
>> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
>> >
>> > > Not sure I understood what bot statistic reduction means.
>> > >
>> > > As far as I understood, you also mean locating failure automatically
>> with
>> > > re-runs on specific git revisions (hashes). I found it will be a good
>> > > contribution to TC Bot for both flaky and non-flaky failures
>> detection.
>> > E.g
>> > > failure occurred after 10 commits merged to master. How to find out
>> bad
>> > > commit? Automatic location (analog of bisecting, but using TC test)
>> will
>> > be
>> > > really helpful. But it is not a simple contribution.
>> > >
>> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <so...@gmail.com>:
>> > >
>> > > > Hi, Dmitriy,
>> > > >
>> > > > Good, I'll create tickets for this steps.
>> > > >
>> > > > Stable suites are a good idea, but also we need to do some actions
>> > when a
>> > > > flaky test will appear in stable suite of master branch run. To find
>> > out
>> > > a
>> > > > guilty commit, we should run previous commits on empty TeamCity's
>> > queue.
>> > > > This runs will reduce bot's statistics for the latest commits.
>> > > >
>> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
>> > > >
>> > > > > Hi Dmitriy,
>> > > > >
>> > > > > Sounds like a plan ;) I totally agree.
>> > > > > Just one proposal: I would like to avoid hiding any test
>> failures. 2
>> > > > > separate tables named 'Possible Blockers' and 'Other failures'
>> should
>> > > be
>> > > > > completely clear. Comment to PR can contain only first category.
>> > > > >
>> > > > > We had a private discussion with Anton K. and he proposed a very
>> > > > > interesting thing, I would like to bring it here. We can add
>> > > > configuration
>> > > > > into TC bot and mark some tests and some suites as 'Known Stable'
>> It
>> > > > means
>> > > > > that suite or test failure should be considered as a blocker for
>> > merge
>> > > > > every time it fails, even if fail rate is non-zero. Very first
>> list
>> > of
>> > > > such
>> > > > > suites are
>> > > > > - Build Apache Ignite
>> > > > > - License check
>> > > > > And tests:
>> > > > > - .Net API Parity check
>> > > > > Please share your vision.
>> > > > >
>> > > > > Meanwhile, I've updated the TC bot.
>> > > > > - Underlying Apache Ignite DB was refactored to use lower
>> partitions
>> > > > count,
>> > > > > so restart should be faster.
>> > > > > - Now 100 builds are saved as our failure rate statistics basis.
>> > > > Flakiness
>> > > > > border was not changed, so more test now will be considered as
>> flaky.
>> > > > > - Now 4 builds required to be failed or timed out in a row before
>> > > > > notification is sent.
>> > > > >
>> > > > > Sincerely,
>> > > > > Dmitriy Pavlov
>> > > > >
>> > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <
>> somefireone@gmail.com>:
>> > > > >
>> > > > > > Hi, Dmitriy,
>> > > > > >
>> > > > > > I propose the next steps:
>> > > > > >
>> > > > > > 1. Show current 0% tests in a separate table at the top of the
>> > > analysis
>> > > > > > results page. Thus, we'll see most suspicious (new or very
>> flaky)
>> > > tests
>> > > > > > firstly. Or we can hide other tests under "More >>" button, like
>> > top
>> > > > long
>> > > > > > running tests.
>> > > > > > 2. Create a button by clicking on which the info about 0% tests
>> > will
>> > > be
>> > > > > > written in the PR.
>> > > > > > 3. Replace button by webhook for TeamCity (for Run All), which
>> will
>> > > > start
>> > > > > > analysis on TCH and write results in the PR.
>> > > > > > 4. ...
>> > > > > >
>> > > > > > What do you think?
>> > > > > >
>> > > > > >
>> > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <somefireone@gmail.com
>> >:
>> > > > > >
>> > > > > > > I think we should check not N last runs, but all runs in last
>> N
>> > > days.
>> > > > > > >
>> > > > > > > A simple rule to detect flaky fails by hands - get test
>> history
>> > > > ordered
>> > > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get
>> list
>> > > of
>> > > > > > > failures from TC. We don't need to check successfull runs,
>> > because
>> > > > they
>> > > > > > are
>> > > > > > > uninformative for our needs.
>> > > > > > >
>> > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <
>> dpavlov.spb@gmail.com
>> > >:
>> > > > > > >
>> > > > > > >> Hi Dmitriy,
>> > > > > > >>
>> > > > > > >> The Bot is able to detect a frequent change of test status,
>> but
>> > > > > > currently
>> > > > > > >> only 50 last runs count. Same is true for the failure rate.
>> > > > > > >>
>> > > > > > >> This value can be easily changed to 70 or 100, moreover, the
>> > auto
>> > > > > > >> trigger feature gives us much more builds.
>> > > > > > >>
>> > > > > > >> We can improve these rules. We can add not only status
>> change,
>> > but
>> > > > > > status
>> > > > > > >> change without any code changes. We can somehow save this
>> data
>> > in
>> > > > > > RunStat
>> > > > > > >> class. Let's create a better rule, and later we can code it.
>> > > > > > >>
>> > > > > > >> Sincerely,
>> > > > > > >> Dmitriy Pavlov
>> > > > > > >>
>> > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <
>> > > somefireone@gmail.com
>> > > > >:
>> > > > > > >>
>> > > > > > >> > I think plugin will be more pretty looking, but comments
>> can
>> > > > contain
>> > > > > > any
>> > > > > > >> > information, so they can be more usefull. I agree with your
>> > idea
>> > > > to
>> > > > > > >> create
>> > > > > > >> > bot instead of plugin.
>> > > > > > >> >
>> > > > > > >> > As for fail rate - I'm not sure it is working as you
>> describe.
>> > > > > > >> > I'm looking on my runAll [1]. There is
>> > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated`
>> > > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky
>> and
>> > > > fails
>> > > > > in
>> > > > > > >> > master branch [2].
>> > > > > > >> >
>> > > > > > >> > [1]
>> > > > > > >> >
>> > > > > > >> > https://mtcga.gridgain.com/pr.
>> html?serverId=apache&suiteId=I
>> > > > > > >>
>> > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&
>> action=Latest
>> > > > > > >> > [2]
>> > > > > > >> >
>> > > > > > >> > https://ci.ignite.apache.org/p
>> roject.html?projectId=IgniteTe
>> > > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470
>> > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests
>> > > > > > >> 24Java8=__all_branches__&itemsCount=50
>> > > > > > >> >
>> > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <
>> > > dpavlov.spb@gmail.com
>> > > > >:
>> > > > > > >> >
>> > > > > > >> > > Hi Dmitrii,
>> > > > > > >> > >
>> > > > > > >> > > I'm not sure we're able to install Github apps to Apache
>> > > > mirrors.
>> > > > > > >> > >
>> > > > > > >> > > The simplest solution, what can be as efficient as a
>> plugin,
>> > > is
>> > > > > fake
>> > > > > > >> > MTCGA
>> > > > > > >> > > bot account in Github, which will provide PR comments
>> using
>> > > > Github
>> > > > > > >> > program
>> > > > > > >> > > interface. What do you think?
>> > > > > > >> > >
>> > > > > > >> > > A new test failure can be identified by the Ignite TC
>> Bot by
>> > > > > master
>> > > > > > >> > recent
>> > > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed
>> out
>> > > > > suites.
>> > > > > > >> > >
>> > > > > > >> > > Sincerely,
>> > > > > > >> > > Dmitriy Pavlov
>> > > > > > >> > >
>> > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <
>> > > > > somefireone@gmail.com
>> > > > > > >:
>> > > > > > >> > >
>> > > > > > >> > > > Hello, Igniters!
>> > > > > > >> > > >
>> > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1]
>> – we
>> > > > need
>> > > > > an
>> > > > > > >> easy
>> > > > > > >> > > way
>> > > > > > >> > > > to get list of failed tests that don’t fall in the
>> master
>> > > > > branch.
>> > > > > > >> These
>> > > > > > >> > > > tests should:
>> > > > > > >> > > > * fail in the PR
>> > > > > > >> > > > * not fail in the master
>> > > > > > >> > > > * not be flaky.
>> > > > > > >> > > >
>> > > > > > >> > > > Also, I want to suggest to create a GitHub plugin,
>> which
>> > > will
>> > > > > > >> notify PR
>> > > > > > >> > > if
>> > > > > > >> > > > it has such tests. PR will have a marker, which
>> > > > allows/prohibits
>> > > > > > >> merge.
>> > > > > > >> > > > This marker will be shown near PR conflicts.
>> > > > > > >> > > >
>> > > > > > >> > > > Allowing marker will be shown in case:
>> > > > > > >> > > > * no new fails.
>> > > > > > >> > > >
>> > > > > > >> > > > Prohibiting marker will be shown in cases:
>> > > > > > >> > > > * new fails – tests must be fixed.
>> > > > > > >> > > > * new timed out test suite – suite should be restarted
>> or
>> > > > tests
>> > > > > > >> must be
>> > > > > > >> > > > fixed.
>> > > > > > >> > > > * runAll wasn’t launched – tests must be launched.
>> > > > > > >> > > >
>> > > > > > >> > > > This will make test checks much faster and easier.
>> Also,
>> > > this
>> > > > > will
>> > > > > > >> > > decrease
>> > > > > > >> > > > the number of merges with new failed tests made by
>> > > inattention
>> > > > > to
>> > > > > > >> the
>> > > > > > >> > > > tests.
>> > > > > > >> > > >
>> > > > > > >> > > > Further, we can expand the plugin by adding new checks,
>> > > > showing
>> > > > > PR
>> > > > > > >> > > quality.
>> > > > > > >> > > >
>> > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot
>> > > > > > >> > > >
>> > > > > > >> > >
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Workflow improvement
Posted by Dmitrii Ryabov <so...@gmail.com>.
Hi, Dmitriy,
I made PRs in my fork for test purposes. Real PRs were made to the GitHub
mirror, and one of them is already merged by D. Govorukhin. PR with GitHub
statuses [1] is ready for review. PR with JIRA comment will be ready in a
few days.
[1] https://github.com/apache/ignite-teamcity-bot/pull/5
2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> Hi Dmitrii,
>
> I deployed change with blockers summary of failures at the top of PR result
> page. The Bot is migrating entries now, I hope it will be done in 1-4
> hours.
>
> I noticed your PRs are created in your own fork, not in
> https://github.com/apache/ignite-teamcity-bot
> Could you please create unmerged PR in GitHub mirror so it could be merged
> after reviewing by the apply-pr script.
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <so...@gmail.com>:
>
> > Hi, Dmitriy,
> >
> > I created a table with possible blockers [1] for the page with the latest
> > build analysis. Anton K. has reviewed it. Can you check and merge it?
> >
> > Also, I created a button to notify PR about analisis results [2]. I used
> > GitHub statuses (example [3]), but to work it needs a token with push
> > permission. As Apache PMC, can you acquire such token? I think statuses
> are
> > much more notable than comments.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-9376
> > [2] https://issues.apache.org/jira/browse/IGNITE-9377
> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls
> >
> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> >
> > > Not sure I understood what bot statistic reduction means.
> > >
> > > As far as I understood, you also mean locating failure automatically
> with
> > > re-runs on specific git revisions (hashes). I found it will be a good
> > > contribution to TC Bot for both flaky and non-flaky failures detection.
> > E.g
> > > failure occurred after 10 commits merged to master. How to find out bad
> > > commit? Automatic location (analog of bisecting, but using TC test)
> will
> > be
> > > really helpful. But it is not a simple contribution.
> > >
> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <so...@gmail.com>:
> > >
> > > > Hi, Dmitriy,
> > > >
> > > > Good, I'll create tickets for this steps.
> > > >
> > > > Stable suites are a good idea, but also we need to do some actions
> > when a
> > > > flaky test will appear in stable suite of master branch run. To find
> > out
> > > a
> > > > guilty commit, we should run previous commits on empty TeamCity's
> > queue.
> > > > This runs will reduce bot's statistics for the latest commits.
> > > >
> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> > > >
> > > > > Hi Dmitriy,
> > > > >
> > > > > Sounds like a plan ;) I totally agree.
> > > > > Just one proposal: I would like to avoid hiding any test failures.
> 2
> > > > > separate tables named 'Possible Blockers' and 'Other failures'
> should
> > > be
> > > > > completely clear. Comment to PR can contain only first category.
> > > > >
> > > > > We had a private discussion with Anton K. and he proposed a very
> > > > > interesting thing, I would like to bring it here. We can add
> > > > configuration
> > > > > into TC bot and mark some tests and some suites as 'Known Stable'
> It
> > > > means
> > > > > that suite or test failure should be considered as a blocker for
> > merge
> > > > > every time it fails, even if fail rate is non-zero. Very first list
> > of
> > > > such
> > > > > suites are
> > > > > - Build Apache Ignite
> > > > > - License check
> > > > > And tests:
> > > > > - .Net API Parity check
> > > > > Please share your vision.
> > > > >
> > > > > Meanwhile, I've updated the TC bot.
> > > > > - Underlying Apache Ignite DB was refactored to use lower
> partitions
> > > > count,
> > > > > so restart should be faster.
> > > > > - Now 100 builds are saved as our failure rate statistics basis.
> > > > Flakiness
> > > > > border was not changed, so more test now will be considered as
> flaky.
> > > > > - Now 4 builds required to be failed or timed out in a row before
> > > > > notification is sent.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <somefireone@gmail.com
> >:
> > > > >
> > > > > > Hi, Dmitriy,
> > > > > >
> > > > > > I propose the next steps:
> > > > > >
> > > > > > 1. Show current 0% tests in a separate table at the top of the
> > > analysis
> > > > > > results page. Thus, we'll see most suspicious (new or very flaky)
> > > tests
> > > > > > firstly. Or we can hide other tests under "More >>" button, like
> > top
> > > > long
> > > > > > running tests.
> > > > > > 2. Create a button by clicking on which the info about 0% tests
> > will
> > > be
> > > > > > written in the PR.
> > > > > > 3. Replace button by webhook for TeamCity (for Run All), which
> will
> > > > start
> > > > > > analysis on TCH and write results in the PR.
> > > > > > 4. ...
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > >
> > > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <somefireone@gmail.com
> >:
> > > > > >
> > > > > > > I think we should check not N last runs, but all runs in last N
> > > days.
> > > > > > >
> > > > > > > A simple rule to detect flaky fails by hands - get test history
> > > > ordered
> > > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get
> list
> > > of
> > > > > > > failures from TC. We don't need to check successfull runs,
> > because
> > > > they
> > > > > > are
> > > > > > > uninformative for our needs.
> > > > > > >
> > > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <
> dpavlov.spb@gmail.com
> > >:
> > > > > > >
> > > > > > >> Hi Dmitriy,
> > > > > > >>
> > > > > > >> The Bot is able to detect a frequent change of test status,
> but
> > > > > > currently
> > > > > > >> only 50 last runs count. Same is true for the failure rate.
> > > > > > >>
> > > > > > >> This value can be easily changed to 70 or 100, moreover, the
> > auto
> > > > > > >> trigger feature gives us much more builds.
> > > > > > >>
> > > > > > >> We can improve these rules. We can add not only status change,
> > but
> > > > > > status
> > > > > > >> change without any code changes. We can somehow save this data
> > in
> > > > > > RunStat
> > > > > > >> class. Let's create a better rule, and later we can code it.
> > > > > > >>
> > > > > > >> Sincerely,
> > > > > > >> Dmitriy Pavlov
> > > > > > >>
> > > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <
> > > somefireone@gmail.com
> > > > >:
> > > > > > >>
> > > > > > >> > I think plugin will be more pretty looking, but comments can
> > > > contain
> > > > > > any
> > > > > > >> > information, so they can be more usefull. I agree with your
> > idea
> > > > to
> > > > > > >> create
> > > > > > >> > bot instead of plugin.
> > > > > > >> >
> > > > > > >> > As for fail rate - I'm not sure it is working as you
> describe.
> > > > > > >> > I'm looking on my runAll [1]. There is
> > > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated`
> > > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky
> and
> > > > fails
> > > > > in
> > > > > > >> > master branch [2].
> > > > > > >> >
> > > > > > >> > [1]
> > > > > > >> >
> > > > > > >> > https://mtcga.gridgain.com/pr.
> html?serverId=apache&suiteId=I
> > > > > > >>
> > > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%
> 2Fhead&action=Latest
> > > > > > >> > [2]
> > > > > > >> >
> > > > > > >> > https://ci.ignite.apache.org/project.html?projectId=
> IgniteTe
> > > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470
> > > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests
> > > > > > >> 24Java8=__all_branches__&itemsCount=50
> > > > > > >> >
> > > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <
> > > dpavlov.spb@gmail.com
> > > > >:
> > > > > > >> >
> > > > > > >> > > Hi Dmitrii,
> > > > > > >> > >
> > > > > > >> > > I'm not sure we're able to install Github apps to Apache
> > > > mirrors.
> > > > > > >> > >
> > > > > > >> > > The simplest solution, what can be as efficient as a
> plugin,
> > > is
> > > > > fake
> > > > > > >> > MTCGA
> > > > > > >> > > bot account in Github, which will provide PR comments
> using
> > > > Github
> > > > > > >> > program
> > > > > > >> > > interface. What do you think?
> > > > > > >> > >
> > > > > > >> > > A new test failure can be identified by the Ignite TC Bot
> by
> > > > > master
> > > > > > >> > recent
> > > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed
> out
> > > > > suites.
> > > > > > >> > >
> > > > > > >> > > Sincerely,
> > > > > > >> > > Dmitriy Pavlov
> > > > > > >> > >
> > > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <
> > > > > somefireone@gmail.com
> > > > > > >:
> > > > > > >> > >
> > > > > > >> > > > Hello, Igniters!
> > > > > > >> > > >
> > > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] –
> we
> > > > need
> > > > > an
> > > > > > >> easy
> > > > > > >> > > way
> > > > > > >> > > > to get list of failed tests that don’t fall in the
> master
> > > > > branch.
> > > > > > >> These
> > > > > > >> > > > tests should:
> > > > > > >> > > > * fail in the PR
> > > > > > >> > > > * not fail in the master
> > > > > > >> > > > * not be flaky.
> > > > > > >> > > >
> > > > > > >> > > > Also, I want to suggest to create a GitHub plugin, which
> > > will
> > > > > > >> notify PR
> > > > > > >> > > if
> > > > > > >> > > > it has such tests. PR will have a marker, which
> > > > allows/prohibits
> > > > > > >> merge.
> > > > > > >> > > > This marker will be shown near PR conflicts.
> > > > > > >> > > >
> > > > > > >> > > > Allowing marker will be shown in case:
> > > > > > >> > > > * no new fails.
> > > > > > >> > > >
> > > > > > >> > > > Prohibiting marker will be shown in cases:
> > > > > > >> > > > * new fails – tests must be fixed.
> > > > > > >> > > > * new timed out test suite – suite should be restarted
> or
> > > > tests
> > > > > > >> must be
> > > > > > >> > > > fixed.
> > > > > > >> > > > * runAll wasn’t launched – tests must be launched.
> > > > > > >> > > >
> > > > > > >> > > > This will make test checks much faster and easier. Also,
> > > this
> > > > > will
> > > > > > >> > > decrease
> > > > > > >> > > > the number of merges with new failed tests made by
> > > inattention
> > > > > to
> > > > > > >> the
> > > > > > >> > > > tests.
> > > > > > >> > > >
> > > > > > >> > > > Further, we can expand the plugin by adding new checks,
> > > > showing
> > > > > PR
> > > > > > >> > > quality.
> > > > > > >> > > >
> > > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Workflow improvement
Posted by Dmitriy Pavlov <dp...@gmail.com>.
Hi Dmitrii,
I deployed change with blockers summary of failures at the top of PR result
page. The Bot is migrating entries now, I hope it will be done in 1-4 hours.
I noticed your PRs are created in your own fork, not in
https://github.com/apache/ignite-teamcity-bot
Could you please create unmerged PR in GitHub mirror so it could be merged
after reviewing by the apply-pr script.
Sincerely,
Dmitriy Pavlov
пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov <so...@gmail.com>:
> Hi, Dmitriy,
>
> I created a table with possible blockers [1] for the page with the latest
> build analysis. Anton K. has reviewed it. Can you check and merge it?
>
> Also, I created a button to notify PR about analisis results [2]. I used
> GitHub statuses (example [3]), but to work it needs a token with push
> permission. As Apache PMC, can you acquire such token? I think statuses are
> much more notable than comments.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9376
> [2] https://issues.apache.org/jira/browse/IGNITE-9377
> [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls
>
> 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
>
> > Not sure I understood what bot statistic reduction means.
> >
> > As far as I understood, you also mean locating failure automatically with
> > re-runs on specific git revisions (hashes). I found it will be a good
> > contribution to TC Bot for both flaky and non-flaky failures detection.
> E.g
> > failure occurred after 10 commits merged to master. How to find out bad
> > commit? Automatic location (analog of bisecting, but using TC test) will
> be
> > really helpful. But it is not a simple contribution.
> >
> > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <so...@gmail.com>:
> >
> > > Hi, Dmitriy,
> > >
> > > Good, I'll create tickets for this steps.
> > >
> > > Stable suites are a good idea, but also we need to do some actions
> when a
> > > flaky test will appear in stable suite of master branch run. To find
> out
> > a
> > > guilty commit, we should run previous commits on empty TeamCity's
> queue.
> > > This runs will reduce bot's statistics for the latest commits.
> > >
> > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> > >
> > > > Hi Dmitriy,
> > > >
> > > > Sounds like a plan ;) I totally agree.
> > > > Just one proposal: I would like to avoid hiding any test failures. 2
> > > > separate tables named 'Possible Blockers' and 'Other failures' should
> > be
> > > > completely clear. Comment to PR can contain only first category.
> > > >
> > > > We had a private discussion with Anton K. and he proposed a very
> > > > interesting thing, I would like to bring it here. We can add
> > > configuration
> > > > into TC bot and mark some tests and some suites as 'Known Stable' It
> > > means
> > > > that suite or test failure should be considered as a blocker for
> merge
> > > > every time it fails, even if fail rate is non-zero. Very first list
> of
> > > such
> > > > suites are
> > > > - Build Apache Ignite
> > > > - License check
> > > > And tests:
> > > > - .Net API Parity check
> > > > Please share your vision.
> > > >
> > > > Meanwhile, I've updated the TC bot.
> > > > - Underlying Apache Ignite DB was refactored to use lower partitions
> > > count,
> > > > so restart should be faster.
> > > > - Now 100 builds are saved as our failure rate statistics basis.
> > > Flakiness
> > > > border was not changed, so more test now will be considered as flaky.
> > > > - Now 4 builds required to be failed or timed out in a row before
> > > > notification is sent.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <so...@gmail.com>:
> > > >
> > > > > Hi, Dmitriy,
> > > > >
> > > > > I propose the next steps:
> > > > >
> > > > > 1. Show current 0% tests in a separate table at the top of the
> > analysis
> > > > > results page. Thus, we'll see most suspicious (new or very flaky)
> > tests
> > > > > firstly. Or we can hide other tests under "More >>" button, like
> top
> > > long
> > > > > running tests.
> > > > > 2. Create a button by clicking on which the info about 0% tests
> will
> > be
> > > > > written in the PR.
> > > > > 3. Replace button by webhook for TeamCity (for Run All), which will
> > > start
> > > > > analysis on TCH and write results in the PR.
> > > > > 4. ...
> > > > >
> > > > > What do you think?
> > > > >
> > > > >
> > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <so...@gmail.com>:
> > > > >
> > > > > > I think we should check not N last runs, but all runs in last N
> > days.
> > > > > >
> > > > > > A simple rule to detect flaky fails by hands - get test history
> > > ordered
> > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get list
> > of
> > > > > > failures from TC. We don't need to check successfull runs,
> because
> > > they
> > > > > are
> > > > > > uninformative for our needs.
> > > > > >
> > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <dpavlov.spb@gmail.com
> >:
> > > > > >
> > > > > >> Hi Dmitriy,
> > > > > >>
> > > > > >> The Bot is able to detect a frequent change of test status, but
> > > > > currently
> > > > > >> only 50 last runs count. Same is true for the failure rate.
> > > > > >>
> > > > > >> This value can be easily changed to 70 or 100, moreover, the
> auto
> > > > > >> trigger feature gives us much more builds.
> > > > > >>
> > > > > >> We can improve these rules. We can add not only status change,
> but
> > > > > status
> > > > > >> change without any code changes. We can somehow save this data
> in
> > > > > RunStat
> > > > > >> class. Let's create a better rule, and later we can code it.
> > > > > >>
> > > > > >> Sincerely,
> > > > > >> Dmitriy Pavlov
> > > > > >>
> > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <
> > somefireone@gmail.com
> > > >:
> > > > > >>
> > > > > >> > I think plugin will be more pretty looking, but comments can
> > > contain
> > > > > any
> > > > > >> > information, so they can be more usefull. I agree with your
> idea
> > > to
> > > > > >> create
> > > > > >> > bot instead of plugin.
> > > > > >> >
> > > > > >> > As for fail rate - I'm not sure it is working as you describe.
> > > > > >> > I'm looking on my runAll [1]. There is
> > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated`
> > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and
> > > fails
> > > > in
> > > > > >> > master branch [2].
> > > > > >> >
> > > > > >> > [1]
> > > > > >> >
> > > > > >> > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=I
> > > > > >>
> > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest
> > > > > >> > [2]
> > > > > >> >
> > > > > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTe
> > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470
> > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests
> > > > > >> 24Java8=__all_branches__&itemsCount=50
> > > > > >> >
> > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <
> > dpavlov.spb@gmail.com
> > > >:
> > > > > >> >
> > > > > >> > > Hi Dmitrii,
> > > > > >> > >
> > > > > >> > > I'm not sure we're able to install Github apps to Apache
> > > mirrors.
> > > > > >> > >
> > > > > >> > > The simplest solution, what can be as efficient as a plugin,
> > is
> > > > fake
> > > > > >> > MTCGA
> > > > > >> > > bot account in Github, which will provide PR comments using
> > > Github
> > > > > >> > program
> > > > > >> > > interface. What do you think?
> > > > > >> > >
> > > > > >> > > A new test failure can be identified by the Ignite TC Bot by
> > > > master
> > > > > >> > recent
> > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed out
> > > > suites.
> > > > > >> > >
> > > > > >> > > Sincerely,
> > > > > >> > > Dmitriy Pavlov
> > > > > >> > >
> > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <
> > > > somefireone@gmail.com
> > > > > >:
> > > > > >> > >
> > > > > >> > > > Hello, Igniters!
> > > > > >> > > >
> > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – we
> > > need
> > > > an
> > > > > >> easy
> > > > > >> > > way
> > > > > >> > > > to get list of failed tests that don’t fall in the master
> > > > branch.
> > > > > >> These
> > > > > >> > > > tests should:
> > > > > >> > > > * fail in the PR
> > > > > >> > > > * not fail in the master
> > > > > >> > > > * not be flaky.
> > > > > >> > > >
> > > > > >> > > > Also, I want to suggest to create a GitHub plugin, which
> > will
> > > > > >> notify PR
> > > > > >> > > if
> > > > > >> > > > it has such tests. PR will have a marker, which
> > > allows/prohibits
> > > > > >> merge.
> > > > > >> > > > This marker will be shown near PR conflicts.
> > > > > >> > > >
> > > > > >> > > > Allowing marker will be shown in case:
> > > > > >> > > > * no new fails.
> > > > > >> > > >
> > > > > >> > > > Prohibiting marker will be shown in cases:
> > > > > >> > > > * new fails – tests must be fixed.
> > > > > >> > > > * new timed out test suite – suite should be restarted or
> > > tests
> > > > > >> must be
> > > > > >> > > > fixed.
> > > > > >> > > > * runAll wasn’t launched – tests must be launched.
> > > > > >> > > >
> > > > > >> > > > This will make test checks much faster and easier. Also,
> > this
> > > > will
> > > > > >> > > decrease
> > > > > >> > > > the number of merges with new failed tests made by
> > inattention
> > > > to
> > > > > >> the
> > > > > >> > > > tests.
> > > > > >> > > >
> > > > > >> > > > Further, we can expand the plugin by adding new checks,
> > > showing
> > > > PR
> > > > > >> > > quality.
> > > > > >> > > >
> > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Workflow improvement
Posted by Dmitriy Pavlov <dp...@gmail.com>.
Hi Dmitrii,
Great news. I love the fact that you did these improvements.
I have limited access to the internet so for now I'm not able to review and
merge.
I can do it next week.
I appreciate your effort.
Sincerely,
Dmitriy Pavlov
пн, 3 сент. 2018 г., 18:20 Dmitrii Ryabov <so...@gmail.com>:
> Hi, Dmitriy,
>
> I created a table with possible blockers [1] for the page with the latest
> build analysis. Anton K. has reviewed it. Can you check and merge it?
>
> Also, I created a button to notify PR about analisis results [2]. I used
> GitHub statuses (example [3]), but to work it needs a token with push
> permission. As Apache PMC, can you acquire such token? I think statuses are
> much more notable than comments.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9376
> [2] https://issues.apache.org/jira/browse/IGNITE-9377
> [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls
>
> 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
>
> > Not sure I understood what bot statistic reduction means.
> >
> > As far as I understood, you also mean locating failure automatically with
> > re-runs on specific git revisions (hashes). I found it will be a good
> > contribution to TC Bot for both flaky and non-flaky failures detection.
> E.g
> > failure occurred after 10 commits merged to master. How to find out bad
> > commit? Automatic location (analog of bisecting, but using TC test) will
> be
> > really helpful. But it is not a simple contribution.
> >
> > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov <so...@gmail.com>:
> >
> > > Hi, Dmitriy,
> > >
> > > Good, I'll create tickets for this steps.
> > >
> > > Stable suites are a good idea, but also we need to do some actions
> when a
> > > flaky test will appear in stable suite of master branch run. To find
> out
> > a
> > > guilty commit, we should run previous commits on empty TeamCity's
> queue.
> > > This runs will reduce bot's statistics for the latest commits.
> > >
> > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov <dp...@gmail.com>:
> > >
> > > > Hi Dmitriy,
> > > >
> > > > Sounds like a plan ;) I totally agree.
> > > > Just one proposal: I would like to avoid hiding any test failures. 2
> > > > separate tables named 'Possible Blockers' and 'Other failures' should
> > be
> > > > completely clear. Comment to PR can contain only first category.
> > > >
> > > > We had a private discussion with Anton K. and he proposed a very
> > > > interesting thing, I would like to bring it here. We can add
> > > configuration
> > > > into TC bot and mark some tests and some suites as 'Known Stable' It
> > > means
> > > > that suite or test failure should be considered as a blocker for
> merge
> > > > every time it fails, even if fail rate is non-zero. Very first list
> of
> > > such
> > > > suites are
> > > > - Build Apache Ignite
> > > > - License check
> > > > And tests:
> > > > - .Net API Parity check
> > > > Please share your vision.
> > > >
> > > > Meanwhile, I've updated the TC bot.
> > > > - Underlying Apache Ignite DB was refactored to use lower partitions
> > > count,
> > > > so restart should be faster.
> > > > - Now 100 builds are saved as our failure rate statistics basis.
> > > Flakiness
> > > > border was not changed, so more test now will be considered as flaky.
> > > > - Now 4 builds required to be failed or timed out in a row before
> > > > notification is sent.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov <so...@gmail.com>:
> > > >
> > > > > Hi, Dmitriy,
> > > > >
> > > > > I propose the next steps:
> > > > >
> > > > > 1. Show current 0% tests in a separate table at the top of the
> > analysis
> > > > > results page. Thus, we'll see most suspicious (new or very flaky)
> > tests
> > > > > firstly. Or we can hide other tests under "More >>" button, like
> top
> > > long
> > > > > running tests.
> > > > > 2. Create a button by clicking on which the info about 0% tests
> will
> > be
> > > > > written in the PR.
> > > > > 3. Replace button by webhook for TeamCity (for Run All), which will
> > > start
> > > > > analysis on TCH and write results in the PR.
> > > > > 4. ...
> > > > >
> > > > > What do you think?
> > > > >
> > > > >
> > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov <so...@gmail.com>:
> > > > >
> > > > > > I think we should check not N last runs, but all runs in last N
> > days.
> > > > > >
> > > > > > A simple rule to detect flaky fails by hands - get test history
> > > ordered
> > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get list
> > of
> > > > > > failures from TC. We don't need to check successfull runs,
> because
> > > they
> > > > > are
> > > > > > uninformative for our needs.
> > > > > >
> > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov <dpavlov.spb@gmail.com
> >:
> > > > > >
> > > > > >> Hi Dmitriy,
> > > > > >>
> > > > > >> The Bot is able to detect a frequent change of test status, but
> > > > > currently
> > > > > >> only 50 last runs count. Same is true for the failure rate.
> > > > > >>
> > > > > >> This value can be easily changed to 70 or 100, moreover, the
> auto
> > > > > >> trigger feature gives us much more builds.
> > > > > >>
> > > > > >> We can improve these rules. We can add not only status change,
> but
> > > > > status
> > > > > >> change without any code changes. We can somehow save this data
> in
> > > > > RunStat
> > > > > >> class. Let's create a better rule, and later we can code it.
> > > > > >>
> > > > > >> Sincerely,
> > > > > >> Dmitriy Pavlov
> > > > > >>
> > > > > >> вт, 21 авг. 2018 г. в 19:22, Dmitrii Ryabov <
> > somefireone@gmail.com
> > > >:
> > > > > >>
> > > > > >> > I think plugin will be more pretty looking, but comments can
> > > contain
> > > > > any
> > > > > >> > information, so they can be more usefull. I agree with your
> idea
> > > to
> > > > > >> create
> > > > > >> > bot instead of plugin.
> > > > > >> >
> > > > > >> > As for fail rate - I'm not sure it is working as you describe.
> > > > > >> > I'm looking on my runAll [1]. There is
> > > > > >> > `IgniteCacheGroupsTest.testCacheApiTxReplicated`
> > > > > >> > in `Cache 3` suite with fail rate = 0.0%. But it is flaky and
> > > fails
> > > > in
> > > > > >> > master branch [2].
> > > > > >> >
> > > > > >> > [1]
> > > > > >> >
> > > > > >> > https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=I
> > > > > >>
> > > gniteTests24Java8_RunAll&branchForTc=pull%2F4519%2Fhead&action=Latest
> > > > > >> > [2]
> > > > > >> >
> > > > > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTe
> > > > > >> sts24Java8&buildTypeId=&tab=testDetails&testNameId=5628470
> > > > > >> 782089555961&order=TEST_STATUS_DESC&branch_IgniteTests
> > > > > >> 24Java8=__all_branches__&itemsCount=50
> > > > > >> >
> > > > > >> > 2018-08-21 18:00 GMT+03:00 Dmitriy Pavlov <
> > dpavlov.spb@gmail.com
> > > >:
> > > > > >> >
> > > > > >> > > Hi Dmitrii,
> > > > > >> > >
> > > > > >> > > I'm not sure we're able to install Github apps to Apache
> > > mirrors.
> > > > > >> > >
> > > > > >> > > The simplest solution, what can be as efficient as a plugin,
> > is
> > > > fake
> > > > > >> > MTCGA
> > > > > >> > > bot account in Github, which will provide PR comments using
> > > Github
> > > > > >> > program
> > > > > >> > > interface. What do you think?
> > > > > >> > >
> > > > > >> > > A new test failure can be identified by the Ignite TC Bot by
> > > > master
> > > > > >> > recent
> > > > > >> > > fail rate = 0.0%. The same rule can be applied to timed out
> > > > suites.
> > > > > >> > >
> > > > > >> > > Sincerely,
> > > > > >> > > Dmitriy Pavlov
> > > > > >> > >
> > > > > >> > > вт, 21 авг. 2018 г. в 16:16, Dmitrii Ryabov <
> > > > somefireone@gmail.com
> > > > > >:
> > > > > >> > >
> > > > > >> > > > Hello, Igniters!
> > > > > >> > > >
> > > > > >> > > > I want to suggest improvement for TeamCity Helper [1] – we
> > > need
> > > > an
> > > > > >> easy
> > > > > >> > > way
> > > > > >> > > > to get list of failed tests that don’t fall in the master
> > > > branch.
> > > > > >> These
> > > > > >> > > > tests should:
> > > > > >> > > > * fail in the PR
> > > > > >> > > > * not fail in the master
> > > > > >> > > > * not be flaky.
> > > > > >> > > >
> > > > > >> > > > Also, I want to suggest to create a GitHub plugin, which
> > will
> > > > > >> notify PR
> > > > > >> > > if
> > > > > >> > > > it has such tests. PR will have a marker, which
> > > allows/prohibits
> > > > > >> merge.
> > > > > >> > > > This marker will be shown near PR conflicts.
> > > > > >> > > >
> > > > > >> > > > Allowing marker will be shown in case:
> > > > > >> > > > * no new fails.
> > > > > >> > > >
> > > > > >> > > > Prohibiting marker will be shown in cases:
> > > > > >> > > > * new fails – tests must be fixed.
> > > > > >> > > > * new timed out test suite – suite should be restarted or
> > > tests
> > > > > >> must be
> > > > > >> > > > fixed.
> > > > > >> > > > * runAll wasn’t launched – tests must be launched.
> > > > > >> > > >
> > > > > >> > > > This will make test checks much faster and easier. Also,
> > this
> > > > will
> > > > > >> > > decrease
> > > > > >> > > > the number of merges with new failed tests made by
> > inattention
> > > > to
> > > > > >> the
> > > > > >> > > > tests.
> > > > > >> > > >
> > > > > >> > > > Further, we can expand the plugin by adding new checks,
> > > showing
> > > > PR
> > > > > >> > > quality.
> > > > > >> > > >
> > > > > >> > > > [1] https://github.com/apache/ignite-teamcity-bot
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>