You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by Laimonas Simutis <la...@gmail.com> on 2015/05/16 21:06:50 UTC
Reproducing random test failures
There are three tests that consistently fail on TC but no matter how many
times I try, I can't reproduce it locally. These tests are:
TestFuzzyQuery.TestTieBreaker
http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
TestSimpleExplanations.TestDMQ8
http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
TestTopDocsMerge.TestSort_2
http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
I would fix them if I could reproduce it -- and I am running out of ideas
how to do it. Even if I put them in a loop running hundreds of times, I
can't trigger the failure.
Anyone have any ideas how to go about reproducing it? I am thinking to push
very verbose code in a separate branch that logs the input values / random
values that are used and see what happens. Checking if anyone has any other
suggestions.
Thanks,
Laimis
Re: Reproducing random test failures
Posted by Laimonas Simutis <la...@gmail.com>.
Thanks Wyatt. Yeah I am aware how to do it in code, was just making sure
that I wasn't missing some TC option that would enable me to switch the
builds / verbosity without code pushes. No big deal, I can check in code
and push and get feedback that way.
Looking at the output now.....
On Thu, May 21, 2015 at 11:44 AM, Wyatt Barnett <wy...@gmail.com>
wrote:
> I found a few minutes to play with things.
>
> First, the general technique to switching things to running under RELEASE
> vs DEBUG while keeping VERBOSE output on is to modify line 299 in
> LuceneTestCase.cs to be "true" -- the preprocessor directives there are
> basically tying DEBUG and VERBOSE together.
>
> Output for TestTieBreaker can be found at http://pastebin.com/VXQYF32T
> Output for TestDMQ8 can be found at http://pastebin.com/WYS07KqB
> Output for TestSort_2 can be downloaded from
>
> https://drive.google.com/file/d/0By01KKFIe0LEX1NubW1mRlVMVWM/view?usp=sharing
>
> I hope this helps get to the bottom of this one, let me know if you need
> further assistance.
>
> On Tue, May 19, 2015 at 2:02 PM Wyatt Barnett <wy...@gmail.com>
> wrote:
>
> > The screenshot didn't come through so I'm kind of guessing here but I
> > switched to debug mode which also triggers VERBOSE to get more output.
> >
> > This confirmed one of the things I saw locally earlier -- that some of
> the
> > semantics switching from debug to release (or non-verbose to verbose come
> > to think of it) makes those tests fail for some reason. The other
> challenge
> > is that the output is so verbose that having verbose on makes the 4th
> test
> > run -- I think the Test_Merge tests -- real, real slow. Like 50+ minute
> > test run slow (
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191532&tab=buildResultsDiv&buildTypeId=LuceneNet_LuceneNetCoreFailingTests
> )
> > compared to ~1 minute test runs.
> >
> > Let me see if I can get output without triggering debug mode when I get
> > home, need to get to the right PC to make that happen.
> >
> > On Tue, May 19, 2015 at 11:56 AM Laimonas Simutis <la...@gmail.com>
> > wrote:
> >
> >> Wyatt,
> >>
> >> Would it be possible to pass that in through the configuration? I tried
> >> passing in verbose flag through but did not appear to work. The same
> with
> >> configuration. Here is the properties section I am referring to:
> >>
> >> [image: Inline image 1]
> >>
> >> Do you know if it something that is possible to have for the TC builds?
> >>
> >> On Tue, May 19, 2015 at 10:54 AM, Wyatt Barnett <
> wyatt.barnett@gmail.com>
> >> wrote:
> >>
> >>> Great. One thing just hit me -- would it be better for this to run as
> >>> DEBUG
> >>> rather than RELEASE? I can flip that switch too . . .
> >>>
> >>> On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <la...@gmail.com>
> >>> wrote:
> >>>
> >>> > Thank you! Just kicked off the build. Let's see what it tells us :)
> >>> >
> >>> > On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <
> >>> wyatt.barnett@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Liamonas -- you should be all set, I just added
> >>> > >
> >>> > >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
> >>> > > which runs the core build with focused tests. Please ignore the
> build
> >>> > > number. Test category is more of a general setting for a build so
> >>> there
> >>> > > isn't an easy checkbox to hit.
> >>> > >
> >>> > > If it makes more sense to re-point that at your github repo just
> say
> >>> the
> >>> > > word and I'll make it so.
> >>> > >
> >>> > >
> >>> > > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <laimis@gmail.com
> >
> >>> > wrote:
> >>> > >
> >>> > > > Wyatt,
> >>> > > >
> >>> > > > I have a branch pushed for this named "failingtests", it is now
> >>> > running a
> >>> > > > build on TC. Where does one specify which category of tests to
> >>> run? I
> >>> > see
> >>> > > > in the settings tab "NUnit categories include/exclude" but don't
> >>> see
> >>> > > where
> >>> > > > to actually modify these values. The tests I would like to run
> >>> belong
> >>> > to
> >>> > > > category "Focus" :) Do you know where to change this?
> >>> > > >
> >>> > > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <
> >>> > wyatt.barnett@gmail.com
> >>> > > >
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Good to hear I checked the right box.
> >>> > > > >
> >>> > > > > I'll see what I can pull together when I get home in terms of
> >>> debug
> >>> > > > output.
> >>> > > > > In terms of testing procedure what I was thinking is we make a
> >>> new
> >>> > > > category
> >>> > > > > -- call it "Focus" and then configure a build looking at your
> >>> fork
> >>> > > > > filtering for just those tests. You can then push away, fire
> off
> >>> > remote
> >>> > > > > builds and check the output yourself.
> >>> > > > >
> >>> > > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <
> >>> laimis@gmail.com>
> >>> > > > wrote:
> >>> > > > >
> >>> > > > > > Wyatt,
> >>> > > > > >
> >>> > > > > > I see the new options on TC, thanks for that. I still haven't
> >>> > thought
> >>> > > > > about
> >>> > > > > > how I will go about capturing the failures exactly, but will
> >>> give
> >>> > > you a
> >>> > > > > > shout if I need some help with TC configuration just for
> those
> >>> > runs.
> >>> > > > > >
> >>> > > > > > If you can reproduce any of those test failures locally, do
> you
> >>> > mind
> >>> > > > > > running them in VERBOSE mode (debug build without any other
> >>> changes
> >>> > > > will
> >>> > > > > > do), and emailing the console output that you get? I might be
> >>> too
> >>> > > > > > optimistic, but perhaps something there will stand out.
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > Thanks again!
> >>> > > > > >
> >>> > > > > > Laimis
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <
> >>> > > > wyatt.barnett@gmail.com>
> >>> > > > > > wrote:
> >>> > > > > >
> >>> > > > > > > For TestSort_2 -- It appears to be passing based on data at
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
> >>> > > > > > > ; I am having locally reproducable problems on the others
> >>> though.
> >>> > > > > > >
> >>> > > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <
> >>> > > > > wyatt.barnett@gmail.com>
> >>> > > > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Done -- you should now see a run button when you visit
> >>> > > > > > > >
> >>> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
> >>> > > > > > > >
> >>> > > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <
> >>> > > > laimis@gmail.com>
> >>> > > > > > > > wrote:
> >>> > > > > > > >
> >>> > > > > > > >> Wyatt,
> >>> > > > > > > >>
> >>> > > > > > > >> Could you add me to the lucene.net group on TC? I have
> a
> >>> > login
> >>> > > > > there,
> >>> > > > > > > >> username: laimis.
> >>> > > > > > > >>
> >>> > > > > > > >>
> >>> > > > > > > >> Thanks!
> >>> > > > > > > >>
> >>> > > > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <
> >>> > > > > > wyatt.barnett@gmail.com
> >>> > > > > > > >
> >>> > > > > > > >> wrote:
> >>> > > > > > > >>
> >>> > > > > > > >> > Sounds good Laimis. You will need to setup a login to
> >>> the
> >>> > > > > CodeBetter
> >>> > > > > > > >> > teamcity server and get added to the lucene.net group
> >>> if
> >>> > you
> >>> > > > > > haven't
> >>> > > > > > > >> > already. Let me know if you need help there too.
> >>> > > > > > > >> >
> >>> > > > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <
> >>> > > > > laimis@gmail.com>
> >>> > > > > > > >> wrote:
> >>> > > > > > > >> >
> >>> > > > > > > >> > > Wyatt,
> >>> > > > > > > >> > >
> >>> > > > > > > >> > > Sweet, I will let you know once I have a branch out
> >>> with
> >>> > > > > > additional
> >>> > > > > > > >> > logging
> >>> > > > > > > >> > > and separate category for tests that you can
> >>> configure to
> >>> > > run.
> >>> > > > > > > >> > >
> >>> > > > > > > >> > > Re: release mode, tried that and was able to fix a
> few
> >>> > bugs
> >>> > > > > after
> >>> > > > > > > >> > switching
> >>> > > > > > > >> > > to it. They were in that PR with debug.assert
> >>> changes. Who
> >>> > > > > knows,
> >>> > > > > > > the
> >>> > > > > > > >> > > remaining failures might still be related to that,
> but
> >>> > can't
> >>> > > > > > > >> reproduce it
> >>> > > > > > > >> > > locally.
> >>> > > > > > > >> > >
> >>> > > > > > > >> > > Laimis
> >>> > > > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <
> >>> > > > > wyatt.barnett@gmail.com
> >>> > > > > > >
> >>> > > > > > > >> > wrote:
> >>> > > > > > > >> > >
> >>> > > > > > > >> > > > Sorry about the blank one -- getting used to
> google
> >>> > inbox
> >>> > > > here
> >>> > > > > > > and I
> >>> > > > > > > >> > > > misclicked.
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > > > Anyhow, I have a repro or at least a rhyme and
> >>> reason --
> >>> > > > > > TeamCity
> >>> > > > > > > is
> >>> > > > > > > >> > > > running in release mode and I think we have
> >>> difffering
> >>> > > > > behavior
> >>> > > > > > > >> there.
> >>> > > > > > > >> > If
> >>> > > > > > > >> > > > you switch your copy of visual studio to release
> >>> mode
> >>> > you
> >>> > > > will
> >>> > > > > > get
> >>> > > > > > > >> the
> >>> > > > > > > >> > > same
> >>> > > > > > > >> > > > failures we are seeing in TeamCity. Does that help
> >>> > narrow
> >>> > > it
> >>> > > > > > down
> >>> > > > > > > a
> >>> > > > > > > >> > bit?
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
> >>> > > > > > > >> wyatt.barnett@gmail.com
> >>> > > > > > > >> > >
> >>> > > > > > > >> > > > wrote:
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > > > >
> >>> > > > > > > >> > > > >
> >>> > > > > > > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
> >>> > > > > > > >> > wyatt.barnett@gmail.com
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > > > > wrote:
> >>> > > > > > > >> > > > >
> >>> > > > > > > >> > > > >> I agree with Itamar -- it feels environmental.
> >>> I'll
> >>> > do
> >>> > > > some
> >>> > > > > > > >> digging
> >>> > > > > > > >> > > into
> >>> > > > > > > >> > > > >> the teamcity output but I think the plan of
> >>> setting
> >>> > up
> >>> > > > some
> >>> > > > > > > extra
> >>> > > > > > > >> > > > verbose
> >>> > > > > > > >> > > > >> logging here would make sense. I can set you up
> >>> with
> >>> > a
> >>> > > > > > separate
> >>> > > > > > > >> > build
> >>> > > > > > > >> > > > >> pointed at your fork if that helps -- it will
> >>> keep
> >>> > the
> >>> > > > > > feedback
> >>> > > > > > > >> > cycle
> >>> > > > > > > >> > > > >> tighter. The other thing we could do is
> >>> categorize
> >>> > the
> >>> > > > > tests
> >>> > > > > > > and
> >>> > > > > > > >> > focus
> >>> > > > > > > >> > > > that
> >>> > > > > > > >> > > > >> build at running only that category so you
> don't
> >>> need
> >>> > > to
> >>> > > > > wait
> >>> > > > > > > on
> >>> > > > > > > >> the
> >>> > > > > > > >> > > > whole
> >>> > > > > > > >> > > > >> suite to get responses. Let me know if you want
> >>> me to
> >>> > > > > proceed
> >>> > > > > > > >> there.
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar
> >>> Syn-Hershko <
> >>> > > > > > > >> > > itamar@code972.com
> >>> > > > > > > >> > > > >
> >>> > > > > > > >> > > > >> wrote:
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > > >>> Yes, that would be the best way to do this. On
> >>> Java
> >>> > > > > Lucene,
> >>> > > > > > > the
> >>> > > > > > > >> > > > >>> randomized
> >>> > > > > > > >> > > > >>> tests framework allows you to re-use the
> random
> >>> seed
> >>> > > > > > > associated
> >>> > > > > > > >> > with
> >>> > > > > > > >> > > > the
> >>> > > > > > > >> > > > >>> failure, but we are not there yet. Either
> way, I
> >>> > > suspect
> >>> > > > > > this
> >>> > > > > > > >> to be
> >>> > > > > > > >> > > an
> >>> > > > > > > >> > > > >>> environment issue rather than a code path one.
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > > >>> --
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > > >>> Itamar Syn-Hershko
> >>> > > > > > > >> > > > >>> http://code972.com | @synhershko <
> >>> > > > > > > >> https://twitter.com/synhershko>
> >>> > > > > > > >> > > > >>> Freelance Developer & Consultant
> >>> > > > > > > >> > > > >>> Lucene.NET committer and PMC member
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas
> >>> Simutis <
> >>> > > > > > > >> > laimis@gmail.com
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > > > >>> wrote:
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > > >>> > There are three tests that consistently fail
> >>> on TC
> >>> > > but
> >>> > > > > no
> >>> > > > > > > >> matter
> >>> > > > > > > >> > > how
> >>> > > > > > > >> > > > >>> many
> >>> > > > > > > >> > > > >>> > times I try, I can't reproduce it locally.
> >>> These
> >>> > > tests
> >>> > > > > > are:
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > >
> >>> > > > > > > >> >
> >>> > > > > > > >>
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > >
> >>> > > > > > > >> >
> >>> > > > > > > >>
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > >
> >>> > > > > > > >> >
> >>> > > > > > > >>
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> > I would fix them if I could reproduce it --
> >>> and I
> >>> > am
> >>> > > > > > running
> >>> > > > > > > >> out
> >>> > > > > > > >> > of
> >>> > > > > > > >> > > > >>> ideas
> >>> > > > > > > >> > > > >>> > how to do it. Even if I put them in a loop
> >>> running
> >>> > > > > > hundreds
> >>> > > > > > > of
> >>> > > > > > > >> > > > times, I
> >>> > > > > > > >> > > > >>> > can't trigger the failure.
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> > Anyone have any ideas how to go about
> >>> reproducing
> >>> > > it?
> >>> > > > I
> >>> > > > > am
> >>> > > > > > > >> > thinking
> >>> > > > > > > >> > > > to
> >>> > > > > > > >> > > > >>> push
> >>> > > > > > > >> > > > >>> > very verbose code in a separate branch that
> >>> logs
> >>> > the
> >>> > > > > input
> >>> > > > > > > >> > values /
> >>> > > > > > > >> > > > >>> random
> >>> > > > > > > >> > > > >>> > values that are used and see what happens.
> >>> > Checking
> >>> > > if
> >>> > > > > > > anyone
> >>> > > > > > > >> has
> >>> > > > > > > >> > > any
> >>> > > > > > > >> > > > >>> other
> >>> > > > > > > >> > > > >>> > suggestions.
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> > Thanks,
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> > Laimis
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > >
> >>> > > > > > > >> >
> >>> > > > > > > >>
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
I found a few minutes to play with things.
First, the general technique to switching things to running under RELEASE
vs DEBUG while keeping VERBOSE output on is to modify line 299 in
LuceneTestCase.cs to be "true" -- the preprocessor directives there are
basically tying DEBUG and VERBOSE together.
Output for TestTieBreaker can be found at http://pastebin.com/VXQYF32T
Output for TestDMQ8 can be found at http://pastebin.com/WYS07KqB
Output for TestSort_2 can be downloaded from
https://drive.google.com/file/d/0By01KKFIe0LEX1NubW1mRlVMVWM/view?usp=sharing
I hope this helps get to the bottom of this one, let me know if you need
further assistance.
On Tue, May 19, 2015 at 2:02 PM Wyatt Barnett <wy...@gmail.com>
wrote:
> The screenshot didn't come through so I'm kind of guessing here but I
> switched to debug mode which also triggers VERBOSE to get more output.
>
> This confirmed one of the things I saw locally earlier -- that some of the
> semantics switching from debug to release (or non-verbose to verbose come
> to think of it) makes those tests fail for some reason. The other challenge
> is that the output is so verbose that having verbose on makes the 4th test
> run -- I think the Test_Merge tests -- real, real slow. Like 50+ minute
> test run slow (
> http://teamcity.codebetter.com/viewLog.html?buildId=191532&tab=buildResultsDiv&buildTypeId=LuceneNet_LuceneNetCoreFailingTests)
> compared to ~1 minute test runs.
>
> Let me see if I can get output without triggering debug mode when I get
> home, need to get to the right PC to make that happen.
>
> On Tue, May 19, 2015 at 11:56 AM Laimonas Simutis <la...@gmail.com>
> wrote:
>
>> Wyatt,
>>
>> Would it be possible to pass that in through the configuration? I tried
>> passing in verbose flag through but did not appear to work. The same with
>> configuration. Here is the properties section I am referring to:
>>
>> [image: Inline image 1]
>>
>> Do you know if it something that is possible to have for the TC builds?
>>
>> On Tue, May 19, 2015 at 10:54 AM, Wyatt Barnett <wy...@gmail.com>
>> wrote:
>>
>>> Great. One thing just hit me -- would it be better for this to run as
>>> DEBUG
>>> rather than RELEASE? I can flip that switch too . . .
>>>
>>> On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <la...@gmail.com>
>>> wrote:
>>>
>>> > Thank you! Just kicked off the build. Let's see what it tells us :)
>>> >
>>> > On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <
>>> wyatt.barnett@gmail.com>
>>> > wrote:
>>> >
>>> > > Liamonas -- you should be all set, I just added
>>> > >
>>> > >
>>> >
>>> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
>>> > > which runs the core build with focused tests. Please ignore the build
>>> > > number. Test category is more of a general setting for a build so
>>> there
>>> > > isn't an easy checkbox to hit.
>>> > >
>>> > > If it makes more sense to re-point that at your github repo just say
>>> the
>>> > > word and I'll make it so.
>>> > >
>>> > >
>>> > > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <la...@gmail.com>
>>> > wrote:
>>> > >
>>> > > > Wyatt,
>>> > > >
>>> > > > I have a branch pushed for this named "failingtests", it is now
>>> > running a
>>> > > > build on TC. Where does one specify which category of tests to
>>> run? I
>>> > see
>>> > > > in the settings tab "NUnit categories include/exclude" but don't
>>> see
>>> > > where
>>> > > > to actually modify these values. The tests I would like to run
>>> belong
>>> > to
>>> > > > category "Focus" :) Do you know where to change this?
>>> > > >
>>> > > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <
>>> > wyatt.barnett@gmail.com
>>> > > >
>>> > > > wrote:
>>> > > >
>>> > > > > Good to hear I checked the right box.
>>> > > > >
>>> > > > > I'll see what I can pull together when I get home in terms of
>>> debug
>>> > > > output.
>>> > > > > In terms of testing procedure what I was thinking is we make a
>>> new
>>> > > > category
>>> > > > > -- call it "Focus" and then configure a build looking at your
>>> fork
>>> > > > > filtering for just those tests. You can then push away, fire off
>>> > remote
>>> > > > > builds and check the output yourself.
>>> > > > >
>>> > > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <
>>> laimis@gmail.com>
>>> > > > wrote:
>>> > > > >
>>> > > > > > Wyatt,
>>> > > > > >
>>> > > > > > I see the new options on TC, thanks for that. I still haven't
>>> > thought
>>> > > > > about
>>> > > > > > how I will go about capturing the failures exactly, but will
>>> give
>>> > > you a
>>> > > > > > shout if I need some help with TC configuration just for those
>>> > runs.
>>> > > > > >
>>> > > > > > If you can reproduce any of those test failures locally, do you
>>> > mind
>>> > > > > > running them in VERBOSE mode (debug build without any other
>>> changes
>>> > > > will
>>> > > > > > do), and emailing the console output that you get? I might be
>>> too
>>> > > > > > optimistic, but perhaps something there will stand out.
>>> > > > > >
>>> > > > > >
>>> > > > > > Thanks again!
>>> > > > > >
>>> > > > > > Laimis
>>> > > > > >
>>> > > > > >
>>> > > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <
>>> > > > wyatt.barnett@gmail.com>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > For TestSort_2 -- It appears to be passing based on data at
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
>>> > > > > > > ; I am having locally reproducable problems on the others
>>> though.
>>> > > > > > >
>>> > > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <
>>> > > > > wyatt.barnett@gmail.com>
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Done -- you should now see a run button when you visit
>>> > > > > > > >
>>> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
>>> > > > > > > >
>>> > > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <
>>> > > > laimis@gmail.com>
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > >> Wyatt,
>>> > > > > > > >>
>>> > > > > > > >> Could you add me to the lucene.net group on TC? I have a
>>> > login
>>> > > > > there,
>>> > > > > > > >> username: laimis.
>>> > > > > > > >>
>>> > > > > > > >>
>>> > > > > > > >> Thanks!
>>> > > > > > > >>
>>> > > > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <
>>> > > > > > wyatt.barnett@gmail.com
>>> > > > > > > >
>>> > > > > > > >> wrote:
>>> > > > > > > >>
>>> > > > > > > >> > Sounds good Laimis. You will need to setup a login to
>>> the
>>> > > > > CodeBetter
>>> > > > > > > >> > teamcity server and get added to the lucene.net group
>>> if
>>> > you
>>> > > > > > haven't
>>> > > > > > > >> > already. Let me know if you need help there too.
>>> > > > > > > >> >
>>> > > > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <
>>> > > > > laimis@gmail.com>
>>> > > > > > > >> wrote:
>>> > > > > > > >> >
>>> > > > > > > >> > > Wyatt,
>>> > > > > > > >> > >
>>> > > > > > > >> > > Sweet, I will let you know once I have a branch out
>>> with
>>> > > > > > additional
>>> > > > > > > >> > logging
>>> > > > > > > >> > > and separate category for tests that you can
>>> configure to
>>> > > run.
>>> > > > > > > >> > >
>>> > > > > > > >> > > Re: release mode, tried that and was able to fix a few
>>> > bugs
>>> > > > > after
>>> > > > > > > >> > switching
>>> > > > > > > >> > > to it. They were in that PR with debug.assert
>>> changes. Who
>>> > > > > knows,
>>> > > > > > > the
>>> > > > > > > >> > > remaining failures might still be related to that, but
>>> > can't
>>> > > > > > > >> reproduce it
>>> > > > > > > >> > > locally.
>>> > > > > > > >> > >
>>> > > > > > > >> > > Laimis
>>> > > > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <
>>> > > > > wyatt.barnett@gmail.com
>>> > > > > > >
>>> > > > > > > >> > wrote:
>>> > > > > > > >> > >
>>> > > > > > > >> > > > Sorry about the blank one -- getting used to google
>>> > inbox
>>> > > > here
>>> > > > > > > and I
>>> > > > > > > >> > > > misclicked.
>>> > > > > > > >> > > >
>>> > > > > > > >> > > > Anyhow, I have a repro or at least a rhyme and
>>> reason --
>>> > > > > > TeamCity
>>> > > > > > > is
>>> > > > > > > >> > > > running in release mode and I think we have
>>> difffering
>>> > > > > behavior
>>> > > > > > > >> there.
>>> > > > > > > >> > If
>>> > > > > > > >> > > > you switch your copy of visual studio to release
>>> mode
>>> > you
>>> > > > will
>>> > > > > > get
>>> > > > > > > >> the
>>> > > > > > > >> > > same
>>> > > > > > > >> > > > failures we are seeing in TeamCity. Does that help
>>> > narrow
>>> > > it
>>> > > > > > down
>>> > > > > > > a
>>> > > > > > > >> > bit?
>>> > > > > > > >> > > >
>>> > > > > > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
>>> > > > > > > >> wyatt.barnett@gmail.com
>>> > > > > > > >> > >
>>> > > > > > > >> > > > wrote:
>>> > > > > > > >> > > >
>>> > > > > > > >> > > > >
>>> > > > > > > >> > > > >
>>> > > > > > > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
>>> > > > > > > >> > wyatt.barnett@gmail.com
>>> > > > > > > >> > > >
>>> > > > > > > >> > > > > wrote:
>>> > > > > > > >> > > > >
>>> > > > > > > >> > > > >> I agree with Itamar -- it feels environmental.
>>> I'll
>>> > do
>>> > > > some
>>> > > > > > > >> digging
>>> > > > > > > >> > > into
>>> > > > > > > >> > > > >> the teamcity output but I think the plan of
>>> setting
>>> > up
>>> > > > some
>>> > > > > > > extra
>>> > > > > > > >> > > > verbose
>>> > > > > > > >> > > > >> logging here would make sense. I can set you up
>>> with
>>> > a
>>> > > > > > separate
>>> > > > > > > >> > build
>>> > > > > > > >> > > > >> pointed at your fork if that helps -- it will
>>> keep
>>> > the
>>> > > > > > feedback
>>> > > > > > > >> > cycle
>>> > > > > > > >> > > > >> tighter. The other thing we could do is
>>> categorize
>>> > the
>>> > > > > tests
>>> > > > > > > and
>>> > > > > > > >> > focus
>>> > > > > > > >> > > > that
>>> > > > > > > >> > > > >> build at running only that category so you don't
>>> need
>>> > > to
>>> > > > > wait
>>> > > > > > > on
>>> > > > > > > >> the
>>> > > > > > > >> > > > whole
>>> > > > > > > >> > > > >> suite to get responses. Let me know if you want
>>> me to
>>> > > > > proceed
>>> > > > > > > >> there.
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar
>>> Syn-Hershko <
>>> > > > > > > >> > > itamar@code972.com
>>> > > > > > > >> > > > >
>>> > > > > > > >> > > > >> wrote:
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > > >>> Yes, that would be the best way to do this. On
>>> Java
>>> > > > > Lucene,
>>> > > > > > > the
>>> > > > > > > >> > > > >>> randomized
>>> > > > > > > >> > > > >>> tests framework allows you to re-use the random
>>> seed
>>> > > > > > > associated
>>> > > > > > > >> > with
>>> > > > > > > >> > > > the
>>> > > > > > > >> > > > >>> failure, but we are not there yet. Either way, I
>>> > > suspect
>>> > > > > > this
>>> > > > > > > >> to be
>>> > > > > > > >> > > an
>>> > > > > > > >> > > > >>> environment issue rather than a code path one.
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > > >>> --
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > > >>> Itamar Syn-Hershko
>>> > > > > > > >> > > > >>> http://code972.com | @synhershko <
>>> > > > > > > >> https://twitter.com/synhershko>
>>> > > > > > > >> > > > >>> Freelance Developer & Consultant
>>> > > > > > > >> > > > >>> Lucene.NET committer and PMC member
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas
>>> Simutis <
>>> > > > > > > >> > laimis@gmail.com
>>> > > > > > > >> > > >
>>> > > > > > > >> > > > >>> wrote:
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > > >>> > There are three tests that consistently fail
>>> on TC
>>> > > but
>>> > > > > no
>>> > > > > > > >> matter
>>> > > > > > > >> > > how
>>> > > > > > > >> > > > >>> many
>>> > > > > > > >> > > > >>> > times I try, I can't reproduce it locally.
>>> These
>>> > > tests
>>> > > > > > are:
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > >
>>> > > > > > > >> > >
>>> > > > > > > >> >
>>> > > > > > > >>
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > >
>>> > > > > > > >> > >
>>> > > > > > > >> >
>>> > > > > > > >>
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > >
>>> > > > > > > >> > >
>>> > > > > > > >> >
>>> > > > > > > >>
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > I would fix them if I could reproduce it --
>>> and I
>>> > am
>>> > > > > > running
>>> > > > > > > >> out
>>> > > > > > > >> > of
>>> > > > > > > >> > > > >>> ideas
>>> > > > > > > >> > > > >>> > how to do it. Even if I put them in a loop
>>> running
>>> > > > > > hundreds
>>> > > > > > > of
>>> > > > > > > >> > > > times, I
>>> > > > > > > >> > > > >>> > can't trigger the failure.
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > Anyone have any ideas how to go about
>>> reproducing
>>> > > it?
>>> > > > I
>>> > > > > am
>>> > > > > > > >> > thinking
>>> > > > > > > >> > > > to
>>> > > > > > > >> > > > >>> push
>>> > > > > > > >> > > > >>> > very verbose code in a separate branch that
>>> logs
>>> > the
>>> > > > > input
>>> > > > > > > >> > values /
>>> > > > > > > >> > > > >>> random
>>> > > > > > > >> > > > >>> > values that are used and see what happens.
>>> > Checking
>>> > > if
>>> > > > > > > anyone
>>> > > > > > > >> has
>>> > > > > > > >> > > any
>>> > > > > > > >> > > > >>> other
>>> > > > > > > >> > > > >>> > suggestions.
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > Thanks,
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > Laimis
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > >
>>> > > > > > > >> > >
>>> > > > > > > >> >
>>> > > > > > > >>
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
The screenshot didn't come through so I'm kind of guessing here but I
switched to debug mode which also triggers VERBOSE to get more output.
This confirmed one of the things I saw locally earlier -- that some of the
semantics switching from debug to release (or non-verbose to verbose come
to think of it) makes those tests fail for some reason. The other challenge
is that the output is so verbose that having verbose on makes the 4th test
run -- I think the Test_Merge tests -- real, real slow. Like 50+ minute
test run slow (
http://teamcity.codebetter.com/viewLog.html?buildId=191532&tab=buildResultsDiv&buildTypeId=LuceneNet_LuceneNetCoreFailingTests)
compared to ~1 minute test runs.
Let me see if I can get output without triggering debug mode when I get
home, need to get to the right PC to make that happen.
On Tue, May 19, 2015 at 11:56 AM Laimonas Simutis <la...@gmail.com> wrote:
> Wyatt,
>
> Would it be possible to pass that in through the configuration? I tried
> passing in verbose flag through but did not appear to work. The same with
> configuration. Here is the properties section I am referring to:
>
> [image: Inline image 1]
>
> Do you know if it something that is possible to have for the TC builds?
>
> On Tue, May 19, 2015 at 10:54 AM, Wyatt Barnett <wy...@gmail.com>
> wrote:
>
>> Great. One thing just hit me -- would it be better for this to run as
>> DEBUG
>> rather than RELEASE? I can flip that switch too . . .
>>
>> On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <la...@gmail.com>
>> wrote:
>>
>> > Thank you! Just kicked off the build. Let's see what it tells us :)
>> >
>> > On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <wyatt.barnett@gmail.com
>> >
>> > wrote:
>> >
>> > > Liamonas -- you should be all set, I just added
>> > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
>> > > which runs the core build with focused tests. Please ignore the build
>> > > number. Test category is more of a general setting for a build so
>> there
>> > > isn't an easy checkbox to hit.
>> > >
>> > > If it makes more sense to re-point that at your github repo just say
>> the
>> > > word and I'll make it so.
>> > >
>> > >
>> > > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <la...@gmail.com>
>> > wrote:
>> > >
>> > > > Wyatt,
>> > > >
>> > > > I have a branch pushed for this named "failingtests", it is now
>> > running a
>> > > > build on TC. Where does one specify which category of tests to run?
>> I
>> > see
>> > > > in the settings tab "NUnit categories include/exclude" but don't see
>> > > where
>> > > > to actually modify these values. The tests I would like to run
>> belong
>> > to
>> > > > category "Focus" :) Do you know where to change this?
>> > > >
>> > > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <
>> > wyatt.barnett@gmail.com
>> > > >
>> > > > wrote:
>> > > >
>> > > > > Good to hear I checked the right box.
>> > > > >
>> > > > > I'll see what I can pull together when I get home in terms of
>> debug
>> > > > output.
>> > > > > In terms of testing procedure what I was thinking is we make a new
>> > > > category
>> > > > > -- call it "Focus" and then configure a build looking at your fork
>> > > > > filtering for just those tests. You can then push away, fire off
>> > remote
>> > > > > builds and check the output yourself.
>> > > > >
>> > > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <
>> laimis@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > > Wyatt,
>> > > > > >
>> > > > > > I see the new options on TC, thanks for that. I still haven't
>> > thought
>> > > > > about
>> > > > > > how I will go about capturing the failures exactly, but will
>> give
>> > > you a
>> > > > > > shout if I need some help with TC configuration just for those
>> > runs.
>> > > > > >
>> > > > > > If you can reproduce any of those test failures locally, do you
>> > mind
>> > > > > > running them in VERBOSE mode (debug build without any other
>> changes
>> > > > will
>> > > > > > do), and emailing the console output that you get? I might be
>> too
>> > > > > > optimistic, but perhaps something there will stand out.
>> > > > > >
>> > > > > >
>> > > > > > Thanks again!
>> > > > > >
>> > > > > > Laimis
>> > > > > >
>> > > > > >
>> > > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <
>> > > > wyatt.barnett@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > For TestSort_2 -- It appears to be passing based on data at
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
>> > > > > > > ; I am having locally reproducable problems on the others
>> though.
>> > > > > > >
>> > > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <
>> > > > > wyatt.barnett@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Done -- you should now see a run button when you visit
>> > > > > > > >
>> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
>> > > > > > > >
>> > > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <
>> > > > laimis@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > >> Wyatt,
>> > > > > > > >>
>> > > > > > > >> Could you add me to the lucene.net group on TC? I have a
>> > login
>> > > > > there,
>> > > > > > > >> username: laimis.
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >> Thanks!
>> > > > > > > >>
>> > > > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <
>> > > > > > wyatt.barnett@gmail.com
>> > > > > > > >
>> > > > > > > >> wrote:
>> > > > > > > >>
>> > > > > > > >> > Sounds good Laimis. You will need to setup a login to the
>> > > > > CodeBetter
>> > > > > > > >> > teamcity server and get added to the lucene.net group if
>> > you
>> > > > > > haven't
>> > > > > > > >> > already. Let me know if you need help there too.
>> > > > > > > >> >
>> > > > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <
>> > > > > laimis@gmail.com>
>> > > > > > > >> wrote:
>> > > > > > > >> >
>> > > > > > > >> > > Wyatt,
>> > > > > > > >> > >
>> > > > > > > >> > > Sweet, I will let you know once I have a branch out
>> with
>> > > > > > additional
>> > > > > > > >> > logging
>> > > > > > > >> > > and separate category for tests that you can configure
>> to
>> > > run.
>> > > > > > > >> > >
>> > > > > > > >> > > Re: release mode, tried that and was able to fix a few
>> > bugs
>> > > > > after
>> > > > > > > >> > switching
>> > > > > > > >> > > to it. They were in that PR with debug.assert changes.
>> Who
>> > > > > knows,
>> > > > > > > the
>> > > > > > > >> > > remaining failures might still be related to that, but
>> > can't
>> > > > > > > >> reproduce it
>> > > > > > > >> > > locally.
>> > > > > > > >> > >
>> > > > > > > >> > > Laimis
>> > > > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <
>> > > > > wyatt.barnett@gmail.com
>> > > > > > >
>> > > > > > > >> > wrote:
>> > > > > > > >> > >
>> > > > > > > >> > > > Sorry about the blank one -- getting used to google
>> > inbox
>> > > > here
>> > > > > > > and I
>> > > > > > > >> > > > misclicked.
>> > > > > > > >> > > >
>> > > > > > > >> > > > Anyhow, I have a repro or at least a rhyme and
>> reason --
>> > > > > > TeamCity
>> > > > > > > is
>> > > > > > > >> > > > running in release mode and I think we have
>> difffering
>> > > > > behavior
>> > > > > > > >> there.
>> > > > > > > >> > If
>> > > > > > > >> > > > you switch your copy of visual studio to release mode
>> > you
>> > > > will
>> > > > > > get
>> > > > > > > >> the
>> > > > > > > >> > > same
>> > > > > > > >> > > > failures we are seeing in TeamCity. Does that help
>> > narrow
>> > > it
>> > > > > > down
>> > > > > > > a
>> > > > > > > >> > bit?
>> > > > > > > >> > > >
>> > > > > > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
>> > > > > > > >> wyatt.barnett@gmail.com
>> > > > > > > >> > >
>> > > > > > > >> > > > wrote:
>> > > > > > > >> > > >
>> > > > > > > >> > > > >
>> > > > > > > >> > > > >
>> > > > > > > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
>> > > > > > > >> > wyatt.barnett@gmail.com
>> > > > > > > >> > > >
>> > > > > > > >> > > > > wrote:
>> > > > > > > >> > > > >
>> > > > > > > >> > > > >> I agree with Itamar -- it feels environmental.
>> I'll
>> > do
>> > > > some
>> > > > > > > >> digging
>> > > > > > > >> > > into
>> > > > > > > >> > > > >> the teamcity output but I think the plan of
>> setting
>> > up
>> > > > some
>> > > > > > > extra
>> > > > > > > >> > > > verbose
>> > > > > > > >> > > > >> logging here would make sense. I can set you up
>> with
>> > a
>> > > > > > separate
>> > > > > > > >> > build
>> > > > > > > >> > > > >> pointed at your fork if that helps -- it will keep
>> > the
>> > > > > > feedback
>> > > > > > > >> > cycle
>> > > > > > > >> > > > >> tighter. The other thing we could do is categorize
>> > the
>> > > > > tests
>> > > > > > > and
>> > > > > > > >> > focus
>> > > > > > > >> > > > that
>> > > > > > > >> > > > >> build at running only that category so you don't
>> need
>> > > to
>> > > > > wait
>> > > > > > > on
>> > > > > > > >> the
>> > > > > > > >> > > > whole
>> > > > > > > >> > > > >> suite to get responses. Let me know if you want
>> me to
>> > > > > proceed
>> > > > > > > >> there.
>> > > > > > > >> > > > >>
>> > > > > > > >> > > > >>
>> > > > > > > >> > > > >>
>> > > > > > > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar
>> Syn-Hershko <
>> > > > > > > >> > > itamar@code972.com
>> > > > > > > >> > > > >
>> > > > > > > >> > > > >> wrote:
>> > > > > > > >> > > > >>
>> > > > > > > >> > > > >>> Yes, that would be the best way to do this. On
>> Java
>> > > > > Lucene,
>> > > > > > > the
>> > > > > > > >> > > > >>> randomized
>> > > > > > > >> > > > >>> tests framework allows you to re-use the random
>> seed
>> > > > > > > associated
>> > > > > > > >> > with
>> > > > > > > >> > > > the
>> > > > > > > >> > > > >>> failure, but we are not there yet. Either way, I
>> > > suspect
>> > > > > > this
>> > > > > > > >> to be
>> > > > > > > >> > > an
>> > > > > > > >> > > > >>> environment issue rather than a code path one.
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > > >>> --
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > > >>> Itamar Syn-Hershko
>> > > > > > > >> > > > >>> http://code972.com | @synhershko <
>> > > > > > > >> https://twitter.com/synhershko>
>> > > > > > > >> > > > >>> Freelance Developer & Consultant
>> > > > > > > >> > > > >>> Lucene.NET committer and PMC member
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas
>> Simutis <
>> > > > > > > >> > laimis@gmail.com
>> > > > > > > >> > > >
>> > > > > > > >> > > > >>> wrote:
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > > >>> > There are three tests that consistently fail
>> on TC
>> > > but
>> > > > > no
>> > > > > > > >> matter
>> > > > > > > >> > > how
>> > > > > > > >> > > > >>> many
>> > > > > > > >> > > > >>> > times I try, I can't reproduce it locally.
>> These
>> > > tests
>> > > > > > are:
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > >
>> > > > > > > >> > >
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > >
>> > > > > > > >> > >
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > >
>> > > > > > > >> > >
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > I would fix them if I could reproduce it --
>> and I
>> > am
>> > > > > > running
>> > > > > > > >> out
>> > > > > > > >> > of
>> > > > > > > >> > > > >>> ideas
>> > > > > > > >> > > > >>> > how to do it. Even if I put them in a loop
>> running
>> > > > > > hundreds
>> > > > > > > of
>> > > > > > > >> > > > times, I
>> > > > > > > >> > > > >>> > can't trigger the failure.
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > Anyone have any ideas how to go about
>> reproducing
>> > > it?
>> > > > I
>> > > > > am
>> > > > > > > >> > thinking
>> > > > > > > >> > > > to
>> > > > > > > >> > > > >>> push
>> > > > > > > >> > > > >>> > very verbose code in a separate branch that
>> logs
>> > the
>> > > > > input
>> > > > > > > >> > values /
>> > > > > > > >> > > > >>> random
>> > > > > > > >> > > > >>> > values that are used and see what happens.
>> > Checking
>> > > if
>> > > > > > > anyone
>> > > > > > > >> has
>> > > > > > > >> > > any
>> > > > > > > >> > > > >>> other
>> > > > > > > >> > > > >>> > suggestions.
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > Thanks,
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > Laimis
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > > >>
>> > > > > > > >> > > > >>
>> > > > > > > >> > > >
>> > > > > > > >> > >
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Reproducing random test failures
Posted by Laimonas Simutis <la...@gmail.com>.
Wyatt,
Would it be possible to pass that in through the configuration? I tried
passing in verbose flag through but did not appear to work. The same with
configuration. Here is the properties section I am referring to:
[image: Inline image 1]
Do you know if it something that is possible to have for the TC builds?
On Tue, May 19, 2015 at 10:54 AM, Wyatt Barnett <wy...@gmail.com>
wrote:
> Great. One thing just hit me -- would it be better for this to run as DEBUG
> rather than RELEASE? I can flip that switch too . . .
>
> On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <la...@gmail.com>
> wrote:
>
> > Thank you! Just kicked off the build. Let's see what it tells us :)
> >
> > On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <wy...@gmail.com>
> > wrote:
> >
> > > Liamonas -- you should be all set, I just added
> > >
> > >
> >
> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
> > > which runs the core build with focused tests. Please ignore the build
> > > number. Test category is more of a general setting for a build so there
> > > isn't an easy checkbox to hit.
> > >
> > > If it makes more sense to re-point that at your github repo just say
> the
> > > word and I'll make it so.
> > >
> > >
> > > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <la...@gmail.com>
> > wrote:
> > >
> > > > Wyatt,
> > > >
> > > > I have a branch pushed for this named "failingtests", it is now
> > running a
> > > > build on TC. Where does one specify which category of tests to run? I
> > see
> > > > in the settings tab "NUnit categories include/exclude" but don't see
> > > where
> > > > to actually modify these values. The tests I would like to run belong
> > to
> > > > category "Focus" :) Do you know where to change this?
> > > >
> > > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <
> > wyatt.barnett@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Good to hear I checked the right box.
> > > > >
> > > > > I'll see what I can pull together when I get home in terms of debug
> > > > output.
> > > > > In terms of testing procedure what I was thinking is we make a new
> > > > category
> > > > > -- call it "Focus" and then configure a build looking at your fork
> > > > > filtering for just those tests. You can then push away, fire off
> > remote
> > > > > builds and check the output yourself.
> > > > >
> > > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <laimis@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Wyatt,
> > > > > >
> > > > > > I see the new options on TC, thanks for that. I still haven't
> > thought
> > > > > about
> > > > > > how I will go about capturing the failures exactly, but will give
> > > you a
> > > > > > shout if I need some help with TC configuration just for those
> > runs.
> > > > > >
> > > > > > If you can reproduce any of those test failures locally, do you
> > mind
> > > > > > running them in VERBOSE mode (debug build without any other
> changes
> > > > will
> > > > > > do), and emailing the console output that you get? I might be too
> > > > > > optimistic, but perhaps something there will stand out.
> > > > > >
> > > > > >
> > > > > > Thanks again!
> > > > > >
> > > > > > Laimis
> > > > > >
> > > > > >
> > > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <
> > > > wyatt.barnett@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > For TestSort_2 -- It appears to be passing based on data at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
> > > > > > > ; I am having locally reproducable problems on the others
> though.
> > > > > > >
> > > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <
> > > > > wyatt.barnett@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Done -- you should now see a run button when you visit
> > > > > > > >
> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
> > > > > > > >
> > > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <
> > > > laimis@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Wyatt,
> > > > > > > >>
> > > > > > > >> Could you add me to the lucene.net group on TC? I have a
> > login
> > > > > there,
> > > > > > > >> username: laimis.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Thanks!
> > > > > > > >>
> > > > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <
> > > > > > wyatt.barnett@gmail.com
> > > > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Sounds good Laimis. You will need to setup a login to the
> > > > > CodeBetter
> > > > > > > >> > teamcity server and get added to the lucene.net group if
> > you
> > > > > > haven't
> > > > > > > >> > already. Let me know if you need help there too.
> > > > > > > >> >
> > > > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <
> > > > > laimis@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> > > Wyatt,
> > > > > > > >> > >
> > > > > > > >> > > Sweet, I will let you know once I have a branch out with
> > > > > > additional
> > > > > > > >> > logging
> > > > > > > >> > > and separate category for tests that you can configure
> to
> > > run.
> > > > > > > >> > >
> > > > > > > >> > > Re: release mode, tried that and was able to fix a few
> > bugs
> > > > > after
> > > > > > > >> > switching
> > > > > > > >> > > to it. They were in that PR with debug.assert changes.
> Who
> > > > > knows,
> > > > > > > the
> > > > > > > >> > > remaining failures might still be related to that, but
> > can't
> > > > > > > >> reproduce it
> > > > > > > >> > > locally.
> > > > > > > >> > >
> > > > > > > >> > > Laimis
> > > > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <
> > > > > wyatt.barnett@gmail.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Sorry about the blank one -- getting used to google
> > inbox
> > > > here
> > > > > > > and I
> > > > > > > >> > > > misclicked.
> > > > > > > >> > > >
> > > > > > > >> > > > Anyhow, I have a repro or at least a rhyme and reason
> --
> > > > > > TeamCity
> > > > > > > is
> > > > > > > >> > > > running in release mode and I think we have difffering
> > > > > behavior
> > > > > > > >> there.
> > > > > > > >> > If
> > > > > > > >> > > > you switch your copy of visual studio to release mode
> > you
> > > > will
> > > > > > get
> > > > > > > >> the
> > > > > > > >> > > same
> > > > > > > >> > > > failures we are seeing in TeamCity. Does that help
> > narrow
> > > it
> > > > > > down
> > > > > > > a
> > > > > > > >> > bit?
> > > > > > > >> > > >
> > > > > > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
> > > > > > > >> wyatt.barnett@gmail.com
> > > > > > > >> > >
> > > > > > > >> > > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
> > > > > > > >> > wyatt.barnett@gmail.com
> > > > > > > >> > > >
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > >> I agree with Itamar -- it feels environmental. I'll
> > do
> > > > some
> > > > > > > >> digging
> > > > > > > >> > > into
> > > > > > > >> > > > >> the teamcity output but I think the plan of setting
> > up
> > > > some
> > > > > > > extra
> > > > > > > >> > > > verbose
> > > > > > > >> > > > >> logging here would make sense. I can set you up
> with
> > a
> > > > > > separate
> > > > > > > >> > build
> > > > > > > >> > > > >> pointed at your fork if that helps -- it will keep
> > the
> > > > > > feedback
> > > > > > > >> > cycle
> > > > > > > >> > > > >> tighter. The other thing we could do is categorize
> > the
> > > > > tests
> > > > > > > and
> > > > > > > >> > focus
> > > > > > > >> > > > that
> > > > > > > >> > > > >> build at running only that category so you don't
> need
> > > to
> > > > > wait
> > > > > > > on
> > > > > > > >> the
> > > > > > > >> > > > whole
> > > > > > > >> > > > >> suite to get responses. Let me know if you want me
> to
> > > > > proceed
> > > > > > > >> there.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >>
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar
> Syn-Hershko <
> > > > > > > >> > > itamar@code972.com
> > > > > > > >> > > > >
> > > > > > > >> > > > >> wrote:
> > > > > > > >> > > > >>
> > > > > > > >> > > > >>> Yes, that would be the best way to do this. On
> Java
> > > > > Lucene,
> > > > > > > the
> > > > > > > >> > > > >>> randomized
> > > > > > > >> > > > >>> tests framework allows you to re-use the random
> seed
> > > > > > > associated
> > > > > > > >> > with
> > > > > > > >> > > > the
> > > > > > > >> > > > >>> failure, but we are not there yet. Either way, I
> > > suspect
> > > > > > this
> > > > > > > >> to be
> > > > > > > >> > > an
> > > > > > > >> > > > >>> environment issue rather than a code path one.
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> --
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> Itamar Syn-Hershko
> > > > > > > >> > > > >>> http://code972.com | @synhershko <
> > > > > > > >> https://twitter.com/synhershko>
> > > > > > > >> > > > >>> Freelance Developer & Consultant
> > > > > > > >> > > > >>> Lucene.NET committer and PMC member
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas
> Simutis <
> > > > > > > >> > laimis@gmail.com
> > > > > > > >> > > >
> > > > > > > >> > > > >>> wrote:
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> > There are three tests that consistently fail on
> TC
> > > but
> > > > > no
> > > > > > > >> matter
> > > > > > > >> > > how
> > > > > > > >> > > > >>> many
> > > > > > > >> > > > >>> > times I try, I can't reproduce it locally. These
> > > tests
> > > > > > are:
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> > I would fix them if I could reproduce it -- and
> I
> > am
> > > > > > running
> > > > > > > >> out
> > > > > > > >> > of
> > > > > > > >> > > > >>> ideas
> > > > > > > >> > > > >>> > how to do it. Even if I put them in a loop
> running
> > > > > > hundreds
> > > > > > > of
> > > > > > > >> > > > times, I
> > > > > > > >> > > > >>> > can't trigger the failure.
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> > Anyone have any ideas how to go about
> reproducing
> > > it?
> > > > I
> > > > > am
> > > > > > > >> > thinking
> > > > > > > >> > > > to
> > > > > > > >> > > > >>> push
> > > > > > > >> > > > >>> > very verbose code in a separate branch that logs
> > the
> > > > > input
> > > > > > > >> > values /
> > > > > > > >> > > > >>> random
> > > > > > > >> > > > >>> > values that are used and see what happens.
> > Checking
> > > if
> > > > > > > anyone
> > > > > > > >> has
> > > > > > > >> > > any
> > > > > > > >> > > > >>> other
> > > > > > > >> > > > >>> > suggestions.
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> > Thanks,
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>> > Laimis
> > > > > > > >> > > > >>> >
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
Great. One thing just hit me -- would it be better for this to run as DEBUG
rather than RELEASE? I can flip that switch too . . .
On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <la...@gmail.com> wrote:
> Thank you! Just kicked off the build. Let's see what it tells us :)
>
> On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <wy...@gmail.com>
> wrote:
>
> > Liamonas -- you should be all set, I just added
> >
> >
> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
> > which runs the core build with focused tests. Please ignore the build
> > number. Test category is more of a general setting for a build so there
> > isn't an easy checkbox to hit.
> >
> > If it makes more sense to re-point that at your github repo just say the
> > word and I'll make it so.
> >
> >
> > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <la...@gmail.com>
> wrote:
> >
> > > Wyatt,
> > >
> > > I have a branch pushed for this named "failingtests", it is now
> running a
> > > build on TC. Where does one specify which category of tests to run? I
> see
> > > in the settings tab "NUnit categories include/exclude" but don't see
> > where
> > > to actually modify these values. The tests I would like to run belong
> to
> > > category "Focus" :) Do you know where to change this?
> > >
> > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <
> wyatt.barnett@gmail.com
> > >
> > > wrote:
> > >
> > > > Good to hear I checked the right box.
> > > >
> > > > I'll see what I can pull together when I get home in terms of debug
> > > output.
> > > > In terms of testing procedure what I was thinking is we make a new
> > > category
> > > > -- call it "Focus" and then configure a build looking at your fork
> > > > filtering for just those tests. You can then push away, fire off
> remote
> > > > builds and check the output yourself.
> > > >
> > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <la...@gmail.com>
> > > wrote:
> > > >
> > > > > Wyatt,
> > > > >
> > > > > I see the new options on TC, thanks for that. I still haven't
> thought
> > > > about
> > > > > how I will go about capturing the failures exactly, but will give
> > you a
> > > > > shout if I need some help with TC configuration just for those
> runs.
> > > > >
> > > > > If you can reproduce any of those test failures locally, do you
> mind
> > > > > running them in VERBOSE mode (debug build without any other changes
> > > will
> > > > > do), and emailing the console output that you get? I might be too
> > > > > optimistic, but perhaps something there will stand out.
> > > > >
> > > > >
> > > > > Thanks again!
> > > > >
> > > > > Laimis
> > > > >
> > > > >
> > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <
> > > wyatt.barnett@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > For TestSort_2 -- It appears to be passing based on data at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
> > > > > > ; I am having locally reproducable problems on the others though.
> > > > > >
> > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <
> > > > wyatt.barnett@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Done -- you should now see a run button when you visit
> > > > > > >
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet
> > > > > > >
> > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <
> > > laimis@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Wyatt,
> > > > > > >>
> > > > > > >> Could you add me to the lucene.net group on TC? I have a
> login
> > > > there,
> > > > > > >> username: laimis.
> > > > > > >>
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >>
> > > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <
> > > > > wyatt.barnett@gmail.com
> > > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Sounds good Laimis. You will need to setup a login to the
> > > > CodeBetter
> > > > > > >> > teamcity server and get added to the lucene.net group if
> you
> > > > > haven't
> > > > > > >> > already. Let me know if you need help there too.
> > > > > > >> >
> > > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <
> > > > laimis@gmail.com>
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > Wyatt,
> > > > > > >> > >
> > > > > > >> > > Sweet, I will let you know once I have a branch out with
> > > > > additional
> > > > > > >> > logging
> > > > > > >> > > and separate category for tests that you can configure to
> > run.
> > > > > > >> > >
> > > > > > >> > > Re: release mode, tried that and was able to fix a few
> bugs
> > > > after
> > > > > > >> > switching
> > > > > > >> > > to it. They were in that PR with debug.assert changes. Who
> > > > knows,
> > > > > > the
> > > > > > >> > > remaining failures might still be related to that, but
> can't
> > > > > > >> reproduce it
> > > > > > >> > > locally.
> > > > > > >> > >
> > > > > > >> > > Laimis
> > > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <
> > > > wyatt.barnett@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> > >
> > > > > > >> > > > Sorry about the blank one -- getting used to google
> inbox
> > > here
> > > > > > and I
> > > > > > >> > > > misclicked.
> > > > > > >> > > >
> > > > > > >> > > > Anyhow, I have a repro or at least a rhyme and reason --
> > > > > TeamCity
> > > > > > is
> > > > > > >> > > > running in release mode and I think we have difffering
> > > > behavior
> > > > > > >> there.
> > > > > > >> > If
> > > > > > >> > > > you switch your copy of visual studio to release mode
> you
> > > will
> > > > > get
> > > > > > >> the
> > > > > > >> > > same
> > > > > > >> > > > failures we are seeing in TeamCity. Does that help
> narrow
> > it
> > > > > down
> > > > > > a
> > > > > > >> > bit?
> > > > > > >> > > >
> > > > > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
> > > > > > >> wyatt.barnett@gmail.com
> > > > > > >> > >
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
> > > > > > >> > wyatt.barnett@gmail.com
> > > > > > >> > > >
> > > > > > >> > > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > >> I agree with Itamar -- it feels environmental. I'll
> do
> > > some
> > > > > > >> digging
> > > > > > >> > > into
> > > > > > >> > > > >> the teamcity output but I think the plan of setting
> up
> > > some
> > > > > > extra
> > > > > > >> > > > verbose
> > > > > > >> > > > >> logging here would make sense. I can set you up with
> a
> > > > > separate
> > > > > > >> > build
> > > > > > >> > > > >> pointed at your fork if that helps -- it will keep
> the
> > > > > feedback
> > > > > > >> > cycle
> > > > > > >> > > > >> tighter. The other thing we could do is categorize
> the
> > > > tests
> > > > > > and
> > > > > > >> > focus
> > > > > > >> > > > that
> > > > > > >> > > > >> build at running only that category so you don't need
> > to
> > > > wait
> > > > > > on
> > > > > > >> the
> > > > > > >> > > > whole
> > > > > > >> > > > >> suite to get responses. Let me know if you want me to
> > > > proceed
> > > > > > >> there.
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <
> > > > > > >> > > itamar@code972.com
> > > > > > >> > > > >
> > > > > > >> > > > >> wrote:
> > > > > > >> > > > >>
> > > > > > >> > > > >>> Yes, that would be the best way to do this. On Java
> > > > Lucene,
> > > > > > the
> > > > > > >> > > > >>> randomized
> > > > > > >> > > > >>> tests framework allows you to re-use the random seed
> > > > > > associated
> > > > > > >> > with
> > > > > > >> > > > the
> > > > > > >> > > > >>> failure, but we are not there yet. Either way, I
> > suspect
> > > > > this
> > > > > > >> to be
> > > > > > >> > > an
> > > > > > >> > > > >>> environment issue rather than a code path one.
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> --
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> Itamar Syn-Hershko
> > > > > > >> > > > >>> http://code972.com | @synhershko <
> > > > > > >> https://twitter.com/synhershko>
> > > > > > >> > > > >>> Freelance Developer & Consultant
> > > > > > >> > > > >>> Lucene.NET committer and PMC member
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <
> > > > > > >> > laimis@gmail.com
> > > > > > >> > > >
> > > > > > >> > > > >>> wrote:
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> > There are three tests that consistently fail on TC
> > but
> > > > no
> > > > > > >> matter
> > > > > > >> > > how
> > > > > > >> > > > >>> many
> > > > > > >> > > > >>> > times I try, I can't reproduce it locally. These
> > tests
> > > > > are:
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > I would fix them if I could reproduce it -- and I
> am
> > > > > running
> > > > > > >> out
> > > > > > >> > of
> > > > > > >> > > > >>> ideas
> > > > > > >> > > > >>> > how to do it. Even if I put them in a loop running
> > > > > hundreds
> > > > > > of
> > > > > > >> > > > times, I
> > > > > > >> > > > >>> > can't trigger the failure.
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > Anyone have any ideas how to go about reproducing
> > it?
> > > I
> > > > am
> > > > > > >> > thinking
> > > > > > >> > > > to
> > > > > > >> > > > >>> push
> > > > > > >> > > > >>> > very verbose code in a separate branch that logs
> the
> > > > input
> > > > > > >> > values /
> > > > > > >> > > > >>> random
> > > > > > >> > > > >>> > values that are used and see what happens.
> Checking
> > if
> > > > > > anyone
> > > > > > >> has
> > > > > > >> > > any
> > > > > > >> > > > >>> other
> > > > > > >> > > > >>> > suggestions.
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > Thanks,
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > Laimis
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>>
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Reproducing random test failures
Posted by Laimonas Simutis <la...@gmail.com>.
Thank you! Just kicked off the build. Let's see what it tells us :)
On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <wy...@gmail.com>
wrote:
> Liamonas -- you should be all set, I just added
>
> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
> which runs the core build with focused tests. Please ignore the build
> number. Test category is more of a general setting for a build so there
> isn't an easy checkbox to hit.
>
> If it makes more sense to re-point that at your github repo just say the
> word and I'll make it so.
>
>
> On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <la...@gmail.com> wrote:
>
> > Wyatt,
> >
> > I have a branch pushed for this named "failingtests", it is now running a
> > build on TC. Where does one specify which category of tests to run? I see
> > in the settings tab "NUnit categories include/exclude" but don't see
> where
> > to actually modify these values. The tests I would like to run belong to
> > category "Focus" :) Do you know where to change this?
> >
> > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <wyatt.barnett@gmail.com
> >
> > wrote:
> >
> > > Good to hear I checked the right box.
> > >
> > > I'll see what I can pull together when I get home in terms of debug
> > output.
> > > In terms of testing procedure what I was thinking is we make a new
> > category
> > > -- call it "Focus" and then configure a build looking at your fork
> > > filtering for just those tests. You can then push away, fire off remote
> > > builds and check the output yourself.
> > >
> > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <la...@gmail.com>
> > wrote:
> > >
> > > > Wyatt,
> > > >
> > > > I see the new options on TC, thanks for that. I still haven't thought
> > > about
> > > > how I will go about capturing the failures exactly, but will give
> you a
> > > > shout if I need some help with TC configuration just for those runs.
> > > >
> > > > If you can reproduce any of those test failures locally, do you mind
> > > > running them in VERBOSE mode (debug build without any other changes
> > will
> > > > do), and emailing the console output that you get? I might be too
> > > > optimistic, but perhaps something there will stand out.
> > > >
> > > >
> > > > Thanks again!
> > > >
> > > > Laimis
> > > >
> > > >
> > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <
> > wyatt.barnett@gmail.com>
> > > > wrote:
> > > >
> > > > > For TestSort_2 -- It appears to be passing based on data at
> > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
> > > > > ; I am having locally reproducable problems on the others though.
> > > > >
> > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <
> > > wyatt.barnett@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Done -- you should now see a run button when you visit
> > > > > > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
> > > > > >
> > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <
> > laimis@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Wyatt,
> > > > > >>
> > > > > >> Could you add me to the lucene.net group on TC? I have a login
> > > there,
> > > > > >> username: laimis.
> > > > > >>
> > > > > >>
> > > > > >> Thanks!
> > > > > >>
> > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <
> > > > wyatt.barnett@gmail.com
> > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Sounds good Laimis. You will need to setup a login to the
> > > CodeBetter
> > > > > >> > teamcity server and get added to the lucene.net group if you
> > > > haven't
> > > > > >> > already. Let me know if you need help there too.
> > > > > >> >
> > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <
> > > laimis@gmail.com>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Wyatt,
> > > > > >> > >
> > > > > >> > > Sweet, I will let you know once I have a branch out with
> > > > additional
> > > > > >> > logging
> > > > > >> > > and separate category for tests that you can configure to
> run.
> > > > > >> > >
> > > > > >> > > Re: release mode, tried that and was able to fix a few bugs
> > > after
> > > > > >> > switching
> > > > > >> > > to it. They were in that PR with debug.assert changes. Who
> > > knows,
> > > > > the
> > > > > >> > > remaining failures might still be related to that, but can't
> > > > > >> reproduce it
> > > > > >> > > locally.
> > > > > >> > >
> > > > > >> > > Laimis
> > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <
> > > wyatt.barnett@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > > > Sorry about the blank one -- getting used to google inbox
> > here
> > > > > and I
> > > > > >> > > > misclicked.
> > > > > >> > > >
> > > > > >> > > > Anyhow, I have a repro or at least a rhyme and reason --
> > > > TeamCity
> > > > > is
> > > > > >> > > > running in release mode and I think we have difffering
> > > behavior
> > > > > >> there.
> > > > > >> > If
> > > > > >> > > > you switch your copy of visual studio to release mode you
> > will
> > > > get
> > > > > >> the
> > > > > >> > > same
> > > > > >> > > > failures we are seeing in TeamCity. Does that help narrow
> it
> > > > down
> > > > > a
> > > > > >> > bit?
> > > > > >> > > >
> > > > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
> > > > > >> wyatt.barnett@gmail.com
> > > > > >> > >
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
> > > > > >> > wyatt.barnett@gmail.com
> > > > > >> > > >
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > >> I agree with Itamar -- it feels environmental. I'll do
> > some
> > > > > >> digging
> > > > > >> > > into
> > > > > >> > > > >> the teamcity output but I think the plan of setting up
> > some
> > > > > extra
> > > > > >> > > > verbose
> > > > > >> > > > >> logging here would make sense. I can set you up with a
> > > > separate
> > > > > >> > build
> > > > > >> > > > >> pointed at your fork if that helps -- it will keep the
> > > > feedback
> > > > > >> > cycle
> > > > > >> > > > >> tighter. The other thing we could do is categorize the
> > > tests
> > > > > and
> > > > > >> > focus
> > > > > >> > > > that
> > > > > >> > > > >> build at running only that category so you don't need
> to
> > > wait
> > > > > on
> > > > > >> the
> > > > > >> > > > whole
> > > > > >> > > > >> suite to get responses. Let me know if you want me to
> > > proceed
> > > > > >> there.
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <
> > > > > >> > > itamar@code972.com
> > > > > >> > > > >
> > > > > >> > > > >> wrote:
> > > > > >> > > > >>
> > > > > >> > > > >>> Yes, that would be the best way to do this. On Java
> > > Lucene,
> > > > > the
> > > > > >> > > > >>> randomized
> > > > > >> > > > >>> tests framework allows you to re-use the random seed
> > > > > associated
> > > > > >> > with
> > > > > >> > > > the
> > > > > >> > > > >>> failure, but we are not there yet. Either way, I
> suspect
> > > > this
> > > > > >> to be
> > > > > >> > > an
> > > > > >> > > > >>> environment issue rather than a code path one.
> > > > > >> > > > >>>
> > > > > >> > > > >>> --
> > > > > >> > > > >>>
> > > > > >> > > > >>> Itamar Syn-Hershko
> > > > > >> > > > >>> http://code972.com | @synhershko <
> > > > > >> https://twitter.com/synhershko>
> > > > > >> > > > >>> Freelance Developer & Consultant
> > > > > >> > > > >>> Lucene.NET committer and PMC member
> > > > > >> > > > >>>
> > > > > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <
> > > > > >> > laimis@gmail.com
> > > > > >> > > >
> > > > > >> > > > >>> wrote:
> > > > > >> > > > >>>
> > > > > >> > > > >>> > There are three tests that consistently fail on TC
> but
> > > no
> > > > > >> matter
> > > > > >> > > how
> > > > > >> > > > >>> many
> > > > > >> > > > >>> > times I try, I can't reproduce it locally. These
> tests
> > > > are:
> > > > > >> > > > >>> >
> > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
> > > > > >> > > > >>> >
> > > > > >> > > > >>> >
> > > > > >> > > > >>>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> > > > > >> > > > >>> >
> > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
> > > > > >> > > > >>> >
> > > > > >> > > > >>> >
> > > > > >> > > > >>>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> > > > > >> > > > >>> >
> > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2
> > > > > >> > > > >>> >
> > > > > >> > > > >>> >
> > > > > >> > > > >>>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> > > > > >> > > > >>> >
> > > > > >> > > > >>> > I would fix them if I could reproduce it -- and I am
> > > > running
> > > > > >> out
> > > > > >> > of
> > > > > >> > > > >>> ideas
> > > > > >> > > > >>> > how to do it. Even if I put them in a loop running
> > > > hundreds
> > > > > of
> > > > > >> > > > times, I
> > > > > >> > > > >>> > can't trigger the failure.
> > > > > >> > > > >>> >
> > > > > >> > > > >>> > Anyone have any ideas how to go about reproducing
> it?
> > I
> > > am
> > > > > >> > thinking
> > > > > >> > > > to
> > > > > >> > > > >>> push
> > > > > >> > > > >>> > very verbose code in a separate branch that logs the
> > > input
> > > > > >> > values /
> > > > > >> > > > >>> random
> > > > > >> > > > >>> > values that are used and see what happens. Checking
> if
> > > > > anyone
> > > > > >> has
> > > > > >> > > any
> > > > > >> > > > >>> other
> > > > > >> > > > >>> > suggestions.
> > > > > >> > > > >>> >
> > > > > >> > > > >>> >
> > > > > >> > > > >>> > Thanks,
> > > > > >> > > > >>> >
> > > > > >> > > > >>> > Laimis
> > > > > >> > > > >>> >
> > > > > >> > > > >>>
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
Liamonas -- you should be all set, I just added
http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
which runs the core build with focused tests. Please ignore the build
number. Test category is more of a general setting for a build so there
isn't an easy checkbox to hit.
If it makes more sense to re-point that at your github repo just say the
word and I'll make it so.
On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <la...@gmail.com> wrote:
> Wyatt,
>
> I have a branch pushed for this named "failingtests", it is now running a
> build on TC. Where does one specify which category of tests to run? I see
> in the settings tab "NUnit categories include/exclude" but don't see where
> to actually modify these values. The tests I would like to run belong to
> category "Focus" :) Do you know where to change this?
>
> On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <wy...@gmail.com>
> wrote:
>
> > Good to hear I checked the right box.
> >
> > I'll see what I can pull together when I get home in terms of debug
> output.
> > In terms of testing procedure what I was thinking is we make a new
> category
> > -- call it "Focus" and then configure a build looking at your fork
> > filtering for just those tests. You can then push away, fire off remote
> > builds and check the output yourself.
> >
> > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <la...@gmail.com>
> wrote:
> >
> > > Wyatt,
> > >
> > > I see the new options on TC, thanks for that. I still haven't thought
> > about
> > > how I will go about capturing the failures exactly, but will give you a
> > > shout if I need some help with TC configuration just for those runs.
> > >
> > > If you can reproduce any of those test failures locally, do you mind
> > > running them in VERBOSE mode (debug build without any other changes
> will
> > > do), and emailing the console output that you get? I might be too
> > > optimistic, but perhaps something there will stand out.
> > >
> > >
> > > Thanks again!
> > >
> > > Laimis
> > >
> > >
> > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <
> wyatt.barnett@gmail.com>
> > > wrote:
> > >
> > > > For TestSort_2 -- It appears to be passing based on data at
> > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
> > > > ; I am having locally reproducable problems on the others though.
> > > >
> > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <
> > wyatt.barnett@gmail.com>
> > > > wrote:
> > > >
> > > > > Done -- you should now see a run button when you visit
> > > > > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
> > > > >
> > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <
> laimis@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Wyatt,
> > > > >>
> > > > >> Could you add me to the lucene.net group on TC? I have a login
> > there,
> > > > >> username: laimis.
> > > > >>
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <
> > > wyatt.barnett@gmail.com
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Sounds good Laimis. You will need to setup a login to the
> > CodeBetter
> > > > >> > teamcity server and get added to the lucene.net group if you
> > > haven't
> > > > >> > already. Let me know if you need help there too.
> > > > >> >
> > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <
> > laimis@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Wyatt,
> > > > >> > >
> > > > >> > > Sweet, I will let you know once I have a branch out with
> > > additional
> > > > >> > logging
> > > > >> > > and separate category for tests that you can configure to run.
> > > > >> > >
> > > > >> > > Re: release mode, tried that and was able to fix a few bugs
> > after
> > > > >> > switching
> > > > >> > > to it. They were in that PR with debug.assert changes. Who
> > knows,
> > > > the
> > > > >> > > remaining failures might still be related to that, but can't
> > > > >> reproduce it
> > > > >> > > locally.
> > > > >> > >
> > > > >> > > Laimis
> > > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <
> > wyatt.barnett@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > > Sorry about the blank one -- getting used to google inbox
> here
> > > > and I
> > > > >> > > > misclicked.
> > > > >> > > >
> > > > >> > > > Anyhow, I have a repro or at least a rhyme and reason --
> > > TeamCity
> > > > is
> > > > >> > > > running in release mode and I think we have difffering
> > behavior
> > > > >> there.
> > > > >> > If
> > > > >> > > > you switch your copy of visual studio to release mode you
> will
> > > get
> > > > >> the
> > > > >> > > same
> > > > >> > > > failures we are seeing in TeamCity. Does that help narrow it
> > > down
> > > > a
> > > > >> > bit?
> > > > >> > > >
> > > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
> > > > >> wyatt.barnett@gmail.com
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
> > > > >> > wyatt.barnett@gmail.com
> > > > >> > > >
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > >> I agree with Itamar -- it feels environmental. I'll do
> some
> > > > >> digging
> > > > >> > > into
> > > > >> > > > >> the teamcity output but I think the plan of setting up
> some
> > > > extra
> > > > >> > > > verbose
> > > > >> > > > >> logging here would make sense. I can set you up with a
> > > separate
> > > > >> > build
> > > > >> > > > >> pointed at your fork if that helps -- it will keep the
> > > feedback
> > > > >> > cycle
> > > > >> > > > >> tighter. The other thing we could do is categorize the
> > tests
> > > > and
> > > > >> > focus
> > > > >> > > > that
> > > > >> > > > >> build at running only that category so you don't need to
> > wait
> > > > on
> > > > >> the
> > > > >> > > > whole
> > > > >> > > > >> suite to get responses. Let me know if you want me to
> > proceed
> > > > >> there.
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <
> > > > >> > > itamar@code972.com
> > > > >> > > > >
> > > > >> > > > >> wrote:
> > > > >> > > > >>
> > > > >> > > > >>> Yes, that would be the best way to do this. On Java
> > Lucene,
> > > > the
> > > > >> > > > >>> randomized
> > > > >> > > > >>> tests framework allows you to re-use the random seed
> > > > associated
> > > > >> > with
> > > > >> > > > the
> > > > >> > > > >>> failure, but we are not there yet. Either way, I suspect
> > > this
> > > > >> to be
> > > > >> > > an
> > > > >> > > > >>> environment issue rather than a code path one.
> > > > >> > > > >>>
> > > > >> > > > >>> --
> > > > >> > > > >>>
> > > > >> > > > >>> Itamar Syn-Hershko
> > > > >> > > > >>> http://code972.com | @synhershko <
> > > > >> https://twitter.com/synhershko>
> > > > >> > > > >>> Freelance Developer & Consultant
> > > > >> > > > >>> Lucene.NET committer and PMC member
> > > > >> > > > >>>
> > > > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <
> > > > >> > laimis@gmail.com
> > > > >> > > >
> > > > >> > > > >>> wrote:
> > > > >> > > > >>>
> > > > >> > > > >>> > There are three tests that consistently fail on TC but
> > no
> > > > >> matter
> > > > >> > > how
> > > > >> > > > >>> many
> > > > >> > > > >>> > times I try, I can't reproduce it locally. These tests
> > > are:
> > > > >> > > > >>> >
> > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
> > > > >> > > > >>> >
> > > > >> > > > >>> >
> > > > >> > > > >>>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> > > > >> > > > >>> >
> > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
> > > > >> > > > >>> >
> > > > >> > > > >>> >
> > > > >> > > > >>>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> > > > >> > > > >>> >
> > > > >> > > > >>> > TestTopDocsMerge.TestSort_2
> > > > >> > > > >>> >
> > > > >> > > > >>> >
> > > > >> > > > >>>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> > > > >> > > > >>> >
> > > > >> > > > >>> > I would fix them if I could reproduce it -- and I am
> > > running
> > > > >> out
> > > > >> > of
> > > > >> > > > >>> ideas
> > > > >> > > > >>> > how to do it. Even if I put them in a loop running
> > > hundreds
> > > > of
> > > > >> > > > times, I
> > > > >> > > > >>> > can't trigger the failure.
> > > > >> > > > >>> >
> > > > >> > > > >>> > Anyone have any ideas how to go about reproducing it?
> I
> > am
> > > > >> > thinking
> > > > >> > > > to
> > > > >> > > > >>> push
> > > > >> > > > >>> > very verbose code in a separate branch that logs the
> > input
> > > > >> > values /
> > > > >> > > > >>> random
> > > > >> > > > >>> > values that are used and see what happens. Checking if
> > > > anyone
> > > > >> has
> > > > >> > > any
> > > > >> > > > >>> other
> > > > >> > > > >>> > suggestions.
> > > > >> > > > >>> >
> > > > >> > > > >>> >
> > > > >> > > > >>> > Thanks,
> > > > >> > > > >>> >
> > > > >> > > > >>> > Laimis
> > > > >> > > > >>> >
> > > > >> > > > >>>
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Reproducing random test failures
Posted by Laimonas Simutis <la...@gmail.com>.
Wyatt,
I have a branch pushed for this named "failingtests", it is now running a
build on TC. Where does one specify which category of tests to run? I see
in the settings tab "NUnit categories include/exclude" but don't see where
to actually modify these values. The tests I would like to run belong to
category "Focus" :) Do you know where to change this?
On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <wy...@gmail.com>
wrote:
> Good to hear I checked the right box.
>
> I'll see what I can pull together when I get home in terms of debug output.
> In terms of testing procedure what I was thinking is we make a new category
> -- call it "Focus" and then configure a build looking at your fork
> filtering for just those tests. You can then push away, fire off remote
> builds and check the output yourself.
>
> On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <la...@gmail.com> wrote:
>
> > Wyatt,
> >
> > I see the new options on TC, thanks for that. I still haven't thought
> about
> > how I will go about capturing the failures exactly, but will give you a
> > shout if I need some help with TC configuration just for those runs.
> >
> > If you can reproduce any of those test failures locally, do you mind
> > running them in VERBOSE mode (debug build without any other changes will
> > do), and emailing the console output that you get? I might be too
> > optimistic, but perhaps something there will stand out.
> >
> >
> > Thanks again!
> >
> > Laimis
> >
> >
> > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <wy...@gmail.com>
> > wrote:
> >
> > > For TestSort_2 -- It appears to be passing based on data at
> > >
> > >
> >
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
> > > ; I am having locally reproducable problems on the others though.
> > >
> > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <
> wyatt.barnett@gmail.com>
> > > wrote:
> > >
> > > > Done -- you should now see a run button when you visit
> > > > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
> > > >
> > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <la...@gmail.com>
> > > > wrote:
> > > >
> > > >> Wyatt,
> > > >>
> > > >> Could you add me to the lucene.net group on TC? I have a login
> there,
> > > >> username: laimis.
> > > >>
> > > >>
> > > >> Thanks!
> > > >>
> > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <
> > wyatt.barnett@gmail.com
> > > >
> > > >> wrote:
> > > >>
> > > >> > Sounds good Laimis. You will need to setup a login to the
> CodeBetter
> > > >> > teamcity server and get added to the lucene.net group if you
> > haven't
> > > >> > already. Let me know if you need help there too.
> > > >> >
> > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <
> laimis@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > Wyatt,
> > > >> > >
> > > >> > > Sweet, I will let you know once I have a branch out with
> > additional
> > > >> > logging
> > > >> > > and separate category for tests that you can configure to run.
> > > >> > >
> > > >> > > Re: release mode, tried that and was able to fix a few bugs
> after
> > > >> > switching
> > > >> > > to it. They were in that PR with debug.assert changes. Who
> knows,
> > > the
> > > >> > > remaining failures might still be related to that, but can't
> > > >> reproduce it
> > > >> > > locally.
> > > >> > >
> > > >> > > Laimis
> > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <
> wyatt.barnett@gmail.com
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > > > Sorry about the blank one -- getting used to google inbox here
> > > and I
> > > >> > > > misclicked.
> > > >> > > >
> > > >> > > > Anyhow, I have a repro or at least a rhyme and reason --
> > TeamCity
> > > is
> > > >> > > > running in release mode and I think we have difffering
> behavior
> > > >> there.
> > > >> > If
> > > >> > > > you switch your copy of visual studio to release mode you will
> > get
> > > >> the
> > > >> > > same
> > > >> > > > failures we are seeing in TeamCity. Does that help narrow it
> > down
> > > a
> > > >> > bit?
> > > >> > > >
> > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
> > > >> wyatt.barnett@gmail.com
> > > >> > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
> > > >> > wyatt.barnett@gmail.com
> > > >> > > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > >> I agree with Itamar -- it feels environmental. I'll do some
> > > >> digging
> > > >> > > into
> > > >> > > > >> the teamcity output but I think the plan of setting up some
> > > extra
> > > >> > > > verbose
> > > >> > > > >> logging here would make sense. I can set you up with a
> > separate
> > > >> > build
> > > >> > > > >> pointed at your fork if that helps -- it will keep the
> > feedback
> > > >> > cycle
> > > >> > > > >> tighter. The other thing we could do is categorize the
> tests
> > > and
> > > >> > focus
> > > >> > > > that
> > > >> > > > >> build at running only that category so you don't need to
> wait
> > > on
> > > >> the
> > > >> > > > whole
> > > >> > > > >> suite to get responses. Let me know if you want me to
> proceed
> > > >> there.
> > > >> > > > >>
> > > >> > > > >>
> > > >> > > > >>
> > > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <
> > > >> > > itamar@code972.com
> > > >> > > > >
> > > >> > > > >> wrote:
> > > >> > > > >>
> > > >> > > > >>> Yes, that would be the best way to do this. On Java
> Lucene,
> > > the
> > > >> > > > >>> randomized
> > > >> > > > >>> tests framework allows you to re-use the random seed
> > > associated
> > > >> > with
> > > >> > > > the
> > > >> > > > >>> failure, but we are not there yet. Either way, I suspect
> > this
> > > >> to be
> > > >> > > an
> > > >> > > > >>> environment issue rather than a code path one.
> > > >> > > > >>>
> > > >> > > > >>> --
> > > >> > > > >>>
> > > >> > > > >>> Itamar Syn-Hershko
> > > >> > > > >>> http://code972.com | @synhershko <
> > > >> https://twitter.com/synhershko>
> > > >> > > > >>> Freelance Developer & Consultant
> > > >> > > > >>> Lucene.NET committer and PMC member
> > > >> > > > >>>
> > > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <
> > > >> > laimis@gmail.com
> > > >> > > >
> > > >> > > > >>> wrote:
> > > >> > > > >>>
> > > >> > > > >>> > There are three tests that consistently fail on TC but
> no
> > > >> matter
> > > >> > > how
> > > >> > > > >>> many
> > > >> > > > >>> > times I try, I can't reproduce it locally. These tests
> > are:
> > > >> > > > >>> >
> > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
> > > >> > > > >>> >
> > > >> > > > >>> >
> > > >> > > > >>>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> > > >> > > > >>> >
> > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
> > > >> > > > >>> >
> > > >> > > > >>> >
> > > >> > > > >>>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> > > >> > > > >>> >
> > > >> > > > >>> > TestTopDocsMerge.TestSort_2
> > > >> > > > >>> >
> > > >> > > > >>> >
> > > >> > > > >>>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> > > >> > > > >>> >
> > > >> > > > >>> > I would fix them if I could reproduce it -- and I am
> > running
> > > >> out
> > > >> > of
> > > >> > > > >>> ideas
> > > >> > > > >>> > how to do it. Even if I put them in a loop running
> > hundreds
> > > of
> > > >> > > > times, I
> > > >> > > > >>> > can't trigger the failure.
> > > >> > > > >>> >
> > > >> > > > >>> > Anyone have any ideas how to go about reproducing it? I
> am
> > > >> > thinking
> > > >> > > > to
> > > >> > > > >>> push
> > > >> > > > >>> > very verbose code in a separate branch that logs the
> input
> > > >> > values /
> > > >> > > > >>> random
> > > >> > > > >>> > values that are used and see what happens. Checking if
> > > anyone
> > > >> has
> > > >> > > any
> > > >> > > > >>> other
> > > >> > > > >>> > suggestions.
> > > >> > > > >>> >
> > > >> > > > >>> >
> > > >> > > > >>> > Thanks,
> > > >> > > > >>> >
> > > >> > > > >>> > Laimis
> > > >> > > > >>> >
> > > >> > > > >>>
> > > >> > > > >>
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
Good to hear I checked the right box.
I'll see what I can pull together when I get home in terms of debug output.
In terms of testing procedure what I was thinking is we make a new category
-- call it "Focus" and then configure a build looking at your fork
filtering for just those tests. You can then push away, fire off remote
builds and check the output yourself.
On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <la...@gmail.com> wrote:
> Wyatt,
>
> I see the new options on TC, thanks for that. I still haven't thought about
> how I will go about capturing the failures exactly, but will give you a
> shout if I need some help with TC configuration just for those runs.
>
> If you can reproduce any of those test failures locally, do you mind
> running them in VERBOSE mode (debug build without any other changes will
> do), and emailing the console output that you get? I might be too
> optimistic, but perhaps something there will stand out.
>
>
> Thanks again!
>
> Laimis
>
>
> On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <wy...@gmail.com>
> wrote:
>
> > For TestSort_2 -- It appears to be passing based on data at
> >
> >
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
> > ; I am having locally reproducable problems on the others though.
> >
> > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <wy...@gmail.com>
> > wrote:
> >
> > > Done -- you should now see a run button when you visit
> > > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
> > >
> > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <la...@gmail.com>
> > > wrote:
> > >
> > >> Wyatt,
> > >>
> > >> Could you add me to the lucene.net group on TC? I have a login there,
> > >> username: laimis.
> > >>
> > >>
> > >> Thanks!
> > >>
> > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <
> wyatt.barnett@gmail.com
> > >
> > >> wrote:
> > >>
> > >> > Sounds good Laimis. You will need to setup a login to the CodeBetter
> > >> > teamcity server and get added to the lucene.net group if you
> haven't
> > >> > already. Let me know if you need help there too.
> > >> >
> > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <la...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Wyatt,
> > >> > >
> > >> > > Sweet, I will let you know once I have a branch out with
> additional
> > >> > logging
> > >> > > and separate category for tests that you can configure to run.
> > >> > >
> > >> > > Re: release mode, tried that and was able to fix a few bugs after
> > >> > switching
> > >> > > to it. They were in that PR with debug.assert changes. Who knows,
> > the
> > >> > > remaining failures might still be related to that, but can't
> > >> reproduce it
> > >> > > locally.
> > >> > >
> > >> > > Laimis
> > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <wyatt.barnett@gmail.com
> >
> > >> > wrote:
> > >> > >
> > >> > > > Sorry about the blank one -- getting used to google inbox here
> > and I
> > >> > > > misclicked.
> > >> > > >
> > >> > > > Anyhow, I have a repro or at least a rhyme and reason --
> TeamCity
> > is
> > >> > > > running in release mode and I think we have difffering behavior
> > >> there.
> > >> > If
> > >> > > > you switch your copy of visual studio to release mode you will
> get
> > >> the
> > >> > > same
> > >> > > > failures we are seeing in TeamCity. Does that help narrow it
> down
> > a
> > >> > bit?
> > >> > > >
> > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
> > >> wyatt.barnett@gmail.com
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
> > >> > wyatt.barnett@gmail.com
> > >> > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > >> I agree with Itamar -- it feels environmental. I'll do some
> > >> digging
> > >> > > into
> > >> > > > >> the teamcity output but I think the plan of setting up some
> > extra
> > >> > > > verbose
> > >> > > > >> logging here would make sense. I can set you up with a
> separate
> > >> > build
> > >> > > > >> pointed at your fork if that helps -- it will keep the
> feedback
> > >> > cycle
> > >> > > > >> tighter. The other thing we could do is categorize the tests
> > and
> > >> > focus
> > >> > > > that
> > >> > > > >> build at running only that category so you don't need to wait
> > on
> > >> the
> > >> > > > whole
> > >> > > > >> suite to get responses. Let me know if you want me to proceed
> > >> there.
> > >> > > > >>
> > >> > > > >>
> > >> > > > >>
> > >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <
> > >> > > itamar@code972.com
> > >> > > > >
> > >> > > > >> wrote:
> > >> > > > >>
> > >> > > > >>> Yes, that would be the best way to do this. On Java Lucene,
> > the
> > >> > > > >>> randomized
> > >> > > > >>> tests framework allows you to re-use the random seed
> > associated
> > >> > with
> > >> > > > the
> > >> > > > >>> failure, but we are not there yet. Either way, I suspect
> this
> > >> to be
> > >> > > an
> > >> > > > >>> environment issue rather than a code path one.
> > >> > > > >>>
> > >> > > > >>> --
> > >> > > > >>>
> > >> > > > >>> Itamar Syn-Hershko
> > >> > > > >>> http://code972.com | @synhershko <
> > >> https://twitter.com/synhershko>
> > >> > > > >>> Freelance Developer & Consultant
> > >> > > > >>> Lucene.NET committer and PMC member
> > >> > > > >>>
> > >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <
> > >> > laimis@gmail.com
> > >> > > >
> > >> > > > >>> wrote:
> > >> > > > >>>
> > >> > > > >>> > There are three tests that consistently fail on TC but no
> > >> matter
> > >> > > how
> > >> > > > >>> many
> > >> > > > >>> > times I try, I can't reproduce it locally. These tests
> are:
> > >> > > > >>> >
> > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
> > >> > > > >>> >
> > >> > > > >>> >
> > >> > > > >>>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> > >> > > > >>> >
> > >> > > > >>> > TestSimpleExplanations.TestDMQ8
> > >> > > > >>> >
> > >> > > > >>> >
> > >> > > > >>>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> > >> > > > >>> >
> > >> > > > >>> > TestTopDocsMerge.TestSort_2
> > >> > > > >>> >
> > >> > > > >>> >
> > >> > > > >>>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> > >> > > > >>> >
> > >> > > > >>> > I would fix them if I could reproduce it -- and I am
> running
> > >> out
> > >> > of
> > >> > > > >>> ideas
> > >> > > > >>> > how to do it. Even if I put them in a loop running
> hundreds
> > of
> > >> > > > times, I
> > >> > > > >>> > can't trigger the failure.
> > >> > > > >>> >
> > >> > > > >>> > Anyone have any ideas how to go about reproducing it? I am
> > >> > thinking
> > >> > > > to
> > >> > > > >>> push
> > >> > > > >>> > very verbose code in a separate branch that logs the input
> > >> > values /
> > >> > > > >>> random
> > >> > > > >>> > values that are used and see what happens. Checking if
> > anyone
> > >> has
> > >> > > any
> > >> > > > >>> other
> > >> > > > >>> > suggestions.
> > >> > > > >>> >
> > >> > > > >>> >
> > >> > > > >>> > Thanks,
> > >> > > > >>> >
> > >> > > > >>> > Laimis
> > >> > > > >>> >
> > >> > > > >>>
> > >> > > > >>
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Reproducing random test failures
Posted by Laimonas Simutis <la...@gmail.com>.
Wyatt,
I see the new options on TC, thanks for that. I still haven't thought about
how I will go about capturing the failures exactly, but will give you a
shout if I need some help with TC configuration just for those runs.
If you can reproduce any of those test failures locally, do you mind
running them in VERBOSE mode (debug build without any other changes will
do), and emailing the console output that you get? I might be too
optimistic, but perhaps something there will stand out.
Thanks again!
Laimis
On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <wy...@gmail.com>
wrote:
> For TestSort_2 -- It appears to be passing based on data at
>
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
> ; I am having locally reproducable problems on the others though.
>
> On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <wy...@gmail.com>
> wrote:
>
> > Done -- you should now see a run button when you visit
> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
> >
> > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <la...@gmail.com>
> > wrote:
> >
> >> Wyatt,
> >>
> >> Could you add me to the lucene.net group on TC? I have a login there,
> >> username: laimis.
> >>
> >>
> >> Thanks!
> >>
> >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <wyatt.barnett@gmail.com
> >
> >> wrote:
> >>
> >> > Sounds good Laimis. You will need to setup a login to the CodeBetter
> >> > teamcity server and get added to the lucene.net group if you haven't
> >> > already. Let me know if you need help there too.
> >> >
> >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <la...@gmail.com>
> >> wrote:
> >> >
> >> > > Wyatt,
> >> > >
> >> > > Sweet, I will let you know once I have a branch out with additional
> >> > logging
> >> > > and separate category for tests that you can configure to run.
> >> > >
> >> > > Re: release mode, tried that and was able to fix a few bugs after
> >> > switching
> >> > > to it. They were in that PR with debug.assert changes. Who knows,
> the
> >> > > remaining failures might still be related to that, but can't
> >> reproduce it
> >> > > locally.
> >> > >
> >> > > Laimis
> >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <wy...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Sorry about the blank one -- getting used to google inbox here
> and I
> >> > > > misclicked.
> >> > > >
> >> > > > Anyhow, I have a repro or at least a rhyme and reason -- TeamCity
> is
> >> > > > running in release mode and I think we have difffering behavior
> >> there.
> >> > If
> >> > > > you switch your copy of visual studio to release mode you will get
> >> the
> >> > > same
> >> > > > failures we are seeing in TeamCity. Does that help narrow it down
> a
> >> > bit?
> >> > > >
> >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
> >> wyatt.barnett@gmail.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
> >> > wyatt.barnett@gmail.com
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > >> I agree with Itamar -- it feels environmental. I'll do some
> >> digging
> >> > > into
> >> > > > >> the teamcity output but I think the plan of setting up some
> extra
> >> > > > verbose
> >> > > > >> logging here would make sense. I can set you up with a separate
> >> > build
> >> > > > >> pointed at your fork if that helps -- it will keep the feedback
> >> > cycle
> >> > > > >> tighter. The other thing we could do is categorize the tests
> and
> >> > focus
> >> > > > that
> >> > > > >> build at running only that category so you don't need to wait
> on
> >> the
> >> > > > whole
> >> > > > >> suite to get responses. Let me know if you want me to proceed
> >> there.
> >> > > > >>
> >> > > > >>
> >> > > > >>
> >> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <
> >> > > itamar@code972.com
> >> > > > >
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >>> Yes, that would be the best way to do this. On Java Lucene,
> the
> >> > > > >>> randomized
> >> > > > >>> tests framework allows you to re-use the random seed
> associated
> >> > with
> >> > > > the
> >> > > > >>> failure, but we are not there yet. Either way, I suspect this
> >> to be
> >> > > an
> >> > > > >>> environment issue rather than a code path one.
> >> > > > >>>
> >> > > > >>> --
> >> > > > >>>
> >> > > > >>> Itamar Syn-Hershko
> >> > > > >>> http://code972.com | @synhershko <
> >> https://twitter.com/synhershko>
> >> > > > >>> Freelance Developer & Consultant
> >> > > > >>> Lucene.NET committer and PMC member
> >> > > > >>>
> >> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <
> >> > laimis@gmail.com
> >> > > >
> >> > > > >>> wrote:
> >> > > > >>>
> >> > > > >>> > There are three tests that consistently fail on TC but no
> >> matter
> >> > > how
> >> > > > >>> many
> >> > > > >>> > times I try, I can't reproduce it locally. These tests are:
> >> > > > >>> >
> >> > > > >>> > TestFuzzyQuery.TestTieBreaker
> >> > > > >>> >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> >> > > > >>> >
> >> > > > >>> > TestSimpleExplanations.TestDMQ8
> >> > > > >>> >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> >> > > > >>> >
> >> > > > >>> > TestTopDocsMerge.TestSort_2
> >> > > > >>> >
> >> > > > >>> >
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> >> > > > >>> >
> >> > > > >>> > I would fix them if I could reproduce it -- and I am running
> >> out
> >> > of
> >> > > > >>> ideas
> >> > > > >>> > how to do it. Even if I put them in a loop running hundreds
> of
> >> > > > times, I
> >> > > > >>> > can't trigger the failure.
> >> > > > >>> >
> >> > > > >>> > Anyone have any ideas how to go about reproducing it? I am
> >> > thinking
> >> > > > to
> >> > > > >>> push
> >> > > > >>> > very verbose code in a separate branch that logs the input
> >> > values /
> >> > > > >>> random
> >> > > > >>> > values that are used and see what happens. Checking if
> anyone
> >> has
> >> > > any
> >> > > > >>> other
> >> > > > >>> > suggestions.
> >> > > > >>> >
> >> > > > >>> >
> >> > > > >>> > Thanks,
> >> > > > >>> >
> >> > > > >>> > Laimis
> >> > > > >>> >
> >> > > > >>>
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
For TestSort_2 -- It appears to be passing based on data at
http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
; I am having locally reproducable problems on the others though.
On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <wy...@gmail.com>
wrote:
> Done -- you should now see a run button when you visit
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet
>
> On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <la...@gmail.com>
> wrote:
>
>> Wyatt,
>>
>> Could you add me to the lucene.net group on TC? I have a login there,
>> username: laimis.
>>
>>
>> Thanks!
>>
>> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <wy...@gmail.com>
>> wrote:
>>
>> > Sounds good Laimis. You will need to setup a login to the CodeBetter
>> > teamcity server and get added to the lucene.net group if you haven't
>> > already. Let me know if you need help there too.
>> >
>> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <la...@gmail.com>
>> wrote:
>> >
>> > > Wyatt,
>> > >
>> > > Sweet, I will let you know once I have a branch out with additional
>> > logging
>> > > and separate category for tests that you can configure to run.
>> > >
>> > > Re: release mode, tried that and was able to fix a few bugs after
>> > switching
>> > > to it. They were in that PR with debug.assert changes. Who knows, the
>> > > remaining failures might still be related to that, but can't
>> reproduce it
>> > > locally.
>> > >
>> > > Laimis
>> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <wy...@gmail.com>
>> > wrote:
>> > >
>> > > > Sorry about the blank one -- getting used to google inbox here and I
>> > > > misclicked.
>> > > >
>> > > > Anyhow, I have a repro or at least a rhyme and reason -- TeamCity is
>> > > > running in release mode and I think we have difffering behavior
>> there.
>> > If
>> > > > you switch your copy of visual studio to release mode you will get
>> the
>> > > same
>> > > > failures we are seeing in TeamCity. Does that help narrow it down a
>> > bit?
>> > > >
>> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
>> wyatt.barnett@gmail.com
>> > >
>> > > > wrote:
>> > > >
>> > > > >
>> > > > >
>> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
>> > wyatt.barnett@gmail.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > >> I agree with Itamar -- it feels environmental. I'll do some
>> digging
>> > > into
>> > > > >> the teamcity output but I think the plan of setting up some extra
>> > > > verbose
>> > > > >> logging here would make sense. I can set you up with a separate
>> > build
>> > > > >> pointed at your fork if that helps -- it will keep the feedback
>> > cycle
>> > > > >> tighter. The other thing we could do is categorize the tests and
>> > focus
>> > > > that
>> > > > >> build at running only that category so you don't need to wait on
>> the
>> > > > whole
>> > > > >> suite to get responses. Let me know if you want me to proceed
>> there.
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <
>> > > itamar@code972.com
>> > > > >
>> > > > >> wrote:
>> > > > >>
>> > > > >>> Yes, that would be the best way to do this. On Java Lucene, the
>> > > > >>> randomized
>> > > > >>> tests framework allows you to re-use the random seed associated
>> > with
>> > > > the
>> > > > >>> failure, but we are not there yet. Either way, I suspect this
>> to be
>> > > an
>> > > > >>> environment issue rather than a code path one.
>> > > > >>>
>> > > > >>> --
>> > > > >>>
>> > > > >>> Itamar Syn-Hershko
>> > > > >>> http://code972.com | @synhershko <
>> https://twitter.com/synhershko>
>> > > > >>> Freelance Developer & Consultant
>> > > > >>> Lucene.NET committer and PMC member
>> > > > >>>
>> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <
>> > laimis@gmail.com
>> > > >
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>> > There are three tests that consistently fail on TC but no
>> matter
>> > > how
>> > > > >>> many
>> > > > >>> > times I try, I can't reproduce it locally. These tests are:
>> > > > >>> >
>> > > > >>> > TestFuzzyQuery.TestTieBreaker
>> > > > >>> >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
>> > > > >>> >
>> > > > >>> > TestSimpleExplanations.TestDMQ8
>> > > > >>> >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
>> > > > >>> >
>> > > > >>> > TestTopDocsMerge.TestSort_2
>> > > > >>> >
>> > > > >>> >
>> > > > >>>
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
>> > > > >>> >
>> > > > >>> > I would fix them if I could reproduce it -- and I am running
>> out
>> > of
>> > > > >>> ideas
>> > > > >>> > how to do it. Even if I put them in a loop running hundreds of
>> > > > times, I
>> > > > >>> > can't trigger the failure.
>> > > > >>> >
>> > > > >>> > Anyone have any ideas how to go about reproducing it? I am
>> > thinking
>> > > > to
>> > > > >>> push
>> > > > >>> > very verbose code in a separate branch that logs the input
>> > values /
>> > > > >>> random
>> > > > >>> > values that are used and see what happens. Checking if anyone
>> has
>> > > any
>> > > > >>> other
>> > > > >>> > suggestions.
>> > > > >>> >
>> > > > >>> >
>> > > > >>> > Thanks,
>> > > > >>> >
>> > > > >>> > Laimis
>> > > > >>> >
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>>
>
>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
Done -- you should now see a run button when you visit
http://teamcity.codebetter.com/project.html?projectId=LuceneNet
On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <la...@gmail.com> wrote:
> Wyatt,
>
> Could you add me to the lucene.net group on TC? I have a login there,
> username: laimis.
>
>
> Thanks!
>
> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <wy...@gmail.com>
> wrote:
>
> > Sounds good Laimis. You will need to setup a login to the CodeBetter
> > teamcity server and get added to the lucene.net group if you haven't
> > already. Let me know if you need help there too.
> >
> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <la...@gmail.com>
> wrote:
> >
> > > Wyatt,
> > >
> > > Sweet, I will let you know once I have a branch out with additional
> > logging
> > > and separate category for tests that you can configure to run.
> > >
> > > Re: release mode, tried that and was able to fix a few bugs after
> > switching
> > > to it. They were in that PR with debug.assert changes. Who knows, the
> > > remaining failures might still be related to that, but can't reproduce
> it
> > > locally.
> > >
> > > Laimis
> > > On May 16, 2015 4:34 PM, "Wyatt Barnett" <wy...@gmail.com>
> > wrote:
> > >
> > > > Sorry about the blank one -- getting used to google inbox here and I
> > > > misclicked.
> > > >
> > > > Anyhow, I have a repro or at least a rhyme and reason -- TeamCity is
> > > > running in release mode and I think we have difffering behavior
> there.
> > If
> > > > you switch your copy of visual studio to release mode you will get
> the
> > > same
> > > > failures we are seeing in TeamCity. Does that help narrow it down a
> > bit?
> > > >
> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <
> wyatt.barnett@gmail.com
> > >
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
> > wyatt.barnett@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> I agree with Itamar -- it feels environmental. I'll do some
> digging
> > > into
> > > > >> the teamcity output but I think the plan of setting up some extra
> > > > verbose
> > > > >> logging here would make sense. I can set you up with a separate
> > build
> > > > >> pointed at your fork if that helps -- it will keep the feedback
> > cycle
> > > > >> tighter. The other thing we could do is categorize the tests and
> > focus
> > > > that
> > > > >> build at running only that category so you don't need to wait on
> the
> > > > whole
> > > > >> suite to get responses. Let me know if you want me to proceed
> there.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <
> > > itamar@code972.com
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >>> Yes, that would be the best way to do this. On Java Lucene, the
> > > > >>> randomized
> > > > >>> tests framework allows you to re-use the random seed associated
> > with
> > > > the
> > > > >>> failure, but we are not there yet. Either way, I suspect this to
> be
> > > an
> > > > >>> environment issue rather than a code path one.
> > > > >>>
> > > > >>> --
> > > > >>>
> > > > >>> Itamar Syn-Hershko
> > > > >>> http://code972.com | @synhershko <https://twitter.com/synhershko
> >
> > > > >>> Freelance Developer & Consultant
> > > > >>> Lucene.NET committer and PMC member
> > > > >>>
> > > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <
> > laimis@gmail.com
> > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>> > There are three tests that consistently fail on TC but no
> matter
> > > how
> > > > >>> many
> > > > >>> > times I try, I can't reproduce it locally. These tests are:
> > > > >>> >
> > > > >>> > TestFuzzyQuery.TestTieBreaker
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> > > > >>> >
> > > > >>> > TestSimpleExplanations.TestDMQ8
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> > > > >>> >
> > > > >>> > TestTopDocsMerge.TestSort_2
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> > > > >>> >
> > > > >>> > I would fix them if I could reproduce it -- and I am running
> out
> > of
> > > > >>> ideas
> > > > >>> > how to do it. Even if I put them in a loop running hundreds of
> > > > times, I
> > > > >>> > can't trigger the failure.
> > > > >>> >
> > > > >>> > Anyone have any ideas how to go about reproducing it? I am
> > thinking
> > > > to
> > > > >>> push
> > > > >>> > very verbose code in a separate branch that logs the input
> > values /
> > > > >>> random
> > > > >>> > values that are used and see what happens. Checking if anyone
> has
> > > any
> > > > >>> other
> > > > >>> > suggestions.
> > > > >>> >
> > > > >>> >
> > > > >>> > Thanks,
> > > > >>> >
> > > > >>> > Laimis
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: Reproducing random test failures
Posted by Laimonas Simutis <la...@gmail.com>.
Wyatt,
Could you add me to the lucene.net group on TC? I have a login there,
username: laimis.
Thanks!
On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <wy...@gmail.com>
wrote:
> Sounds good Laimis. You will need to setup a login to the CodeBetter
> teamcity server and get added to the lucene.net group if you haven't
> already. Let me know if you need help there too.
>
> On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <la...@gmail.com> wrote:
>
> > Wyatt,
> >
> > Sweet, I will let you know once I have a branch out with additional
> logging
> > and separate category for tests that you can configure to run.
> >
> > Re: release mode, tried that and was able to fix a few bugs after
> switching
> > to it. They were in that PR with debug.assert changes. Who knows, the
> > remaining failures might still be related to that, but can't reproduce it
> > locally.
> >
> > Laimis
> > On May 16, 2015 4:34 PM, "Wyatt Barnett" <wy...@gmail.com>
> wrote:
> >
> > > Sorry about the blank one -- getting used to google inbox here and I
> > > misclicked.
> > >
> > > Anyhow, I have a repro or at least a rhyme and reason -- TeamCity is
> > > running in release mode and I think we have difffering behavior there.
> If
> > > you switch your copy of visual studio to release mode you will get the
> > same
> > > failures we are seeing in TeamCity. Does that help narrow it down a
> bit?
> > >
> > > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <wyatt.barnett@gmail.com
> >
> > > wrote:
> > >
> > > >
> > > >
> > > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <
> wyatt.barnett@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> I agree with Itamar -- it feels environmental. I'll do some digging
> > into
> > > >> the teamcity output but I think the plan of setting up some extra
> > > verbose
> > > >> logging here would make sense. I can set you up with a separate
> build
> > > >> pointed at your fork if that helps -- it will keep the feedback
> cycle
> > > >> tighter. The other thing we could do is categorize the tests and
> focus
> > > that
> > > >> build at running only that category so you don't need to wait on the
> > > whole
> > > >> suite to get responses. Let me know if you want me to proceed there.
> > > >>
> > > >>
> > > >>
> > > >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <
> > itamar@code972.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> Yes, that would be the best way to do this. On Java Lucene, the
> > > >>> randomized
> > > >>> tests framework allows you to re-use the random seed associated
> with
> > > the
> > > >>> failure, but we are not there yet. Either way, I suspect this to be
> > an
> > > >>> environment issue rather than a code path one.
> > > >>>
> > > >>> --
> > > >>>
> > > >>> Itamar Syn-Hershko
> > > >>> http://code972.com | @synhershko <https://twitter.com/synhershko>
> > > >>> Freelance Developer & Consultant
> > > >>> Lucene.NET committer and PMC member
> > > >>>
> > > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <
> laimis@gmail.com
> > >
> > > >>> wrote:
> > > >>>
> > > >>> > There are three tests that consistently fail on TC but no matter
> > how
> > > >>> many
> > > >>> > times I try, I can't reproduce it locally. These tests are:
> > > >>> >
> > > >>> > TestFuzzyQuery.TestTieBreaker
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> > > >>> >
> > > >>> > TestSimpleExplanations.TestDMQ8
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> > > >>> >
> > > >>> > TestTopDocsMerge.TestSort_2
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> > > >>> >
> > > >>> > I would fix them if I could reproduce it -- and I am running out
> of
> > > >>> ideas
> > > >>> > how to do it. Even if I put them in a loop running hundreds of
> > > times, I
> > > >>> > can't trigger the failure.
> > > >>> >
> > > >>> > Anyone have any ideas how to go about reproducing it? I am
> thinking
> > > to
> > > >>> push
> > > >>> > very verbose code in a separate branch that logs the input
> values /
> > > >>> random
> > > >>> > values that are used and see what happens. Checking if anyone has
> > any
> > > >>> other
> > > >>> > suggestions.
> > > >>> >
> > > >>> >
> > > >>> > Thanks,
> > > >>> >
> > > >>> > Laimis
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > >
> >
>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
Sounds good Laimis. You will need to setup a login to the CodeBetter
teamcity server and get added to the lucene.net group if you haven't
already. Let me know if you need help there too.
On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis <la...@gmail.com> wrote:
> Wyatt,
>
> Sweet, I will let you know once I have a branch out with additional logging
> and separate category for tests that you can configure to run.
>
> Re: release mode, tried that and was able to fix a few bugs after switching
> to it. They were in that PR with debug.assert changes. Who knows, the
> remaining failures might still be related to that, but can't reproduce it
> locally.
>
> Laimis
> On May 16, 2015 4:34 PM, "Wyatt Barnett" <wy...@gmail.com> wrote:
>
> > Sorry about the blank one -- getting used to google inbox here and I
> > misclicked.
> >
> > Anyhow, I have a repro or at least a rhyme and reason -- TeamCity is
> > running in release mode and I think we have difffering behavior there. If
> > you switch your copy of visual studio to release mode you will get the
> same
> > failures we are seeing in TeamCity. Does that help narrow it down a bit?
> >
> > On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <wy...@gmail.com>
> > wrote:
> >
> > >
> > >
> > > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <wyatt.barnett@gmail.com
> >
> > > wrote:
> > >
> > >> I agree with Itamar -- it feels environmental. I'll do some digging
> into
> > >> the teamcity output but I think the plan of setting up some extra
> > verbose
> > >> logging here would make sense. I can set you up with a separate build
> > >> pointed at your fork if that helps -- it will keep the feedback cycle
> > >> tighter. The other thing we could do is categorize the tests and focus
> > that
> > >> build at running only that category so you don't need to wait on the
> > whole
> > >> suite to get responses. Let me know if you want me to proceed there.
> > >>
> > >>
> > >>
> > >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <
> itamar@code972.com
> > >
> > >> wrote:
> > >>
> > >>> Yes, that would be the best way to do this. On Java Lucene, the
> > >>> randomized
> > >>> tests framework allows you to re-use the random seed associated with
> > the
> > >>> failure, but we are not there yet. Either way, I suspect this to be
> an
> > >>> environment issue rather than a code path one.
> > >>>
> > >>> --
> > >>>
> > >>> Itamar Syn-Hershko
> > >>> http://code972.com | @synhershko <https://twitter.com/synhershko>
> > >>> Freelance Developer & Consultant
> > >>> Lucene.NET committer and PMC member
> > >>>
> > >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <laimis@gmail.com
> >
> > >>> wrote:
> > >>>
> > >>> > There are three tests that consistently fail on TC but no matter
> how
> > >>> many
> > >>> > times I try, I can't reproduce it locally. These tests are:
> > >>> >
> > >>> > TestFuzzyQuery.TestTieBreaker
> > >>> >
> > >>> >
> > >>>
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> > >>> >
> > >>> > TestSimpleExplanations.TestDMQ8
> > >>> >
> > >>> >
> > >>>
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> > >>> >
> > >>> > TestTopDocsMerge.TestSort_2
> > >>> >
> > >>> >
> > >>>
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> > >>> >
> > >>> > I would fix them if I could reproduce it -- and I am running out of
> > >>> ideas
> > >>> > how to do it. Even if I put them in a loop running hundreds of
> > times, I
> > >>> > can't trigger the failure.
> > >>> >
> > >>> > Anyone have any ideas how to go about reproducing it? I am thinking
> > to
> > >>> push
> > >>> > very verbose code in a separate branch that logs the input values /
> > >>> random
> > >>> > values that are used and see what happens. Checking if anyone has
> any
> > >>> other
> > >>> > suggestions.
> > >>> >
> > >>> >
> > >>> > Thanks,
> > >>> >
> > >>> > Laimis
> > >>> >
> > >>>
> > >>
> > >>
> >
>
Re: Reproducing random test failures
Posted by Laimonas Simutis <la...@gmail.com>.
Wyatt,
Sweet, I will let you know once I have a branch out with additional logging
and separate category for tests that you can configure to run.
Re: release mode, tried that and was able to fix a few bugs after switching
to it. They were in that PR with debug.assert changes. Who knows, the
remaining failures might still be related to that, but can't reproduce it
locally.
Laimis
On May 16, 2015 4:34 PM, "Wyatt Barnett" <wy...@gmail.com> wrote:
> Sorry about the blank one -- getting used to google inbox here and I
> misclicked.
>
> Anyhow, I have a repro or at least a rhyme and reason -- TeamCity is
> running in release mode and I think we have difffering behavior there. If
> you switch your copy of visual studio to release mode you will get the same
> failures we are seeing in TeamCity. Does that help narrow it down a bit?
>
> On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <wy...@gmail.com>
> wrote:
>
> >
> >
> > On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <wy...@gmail.com>
> > wrote:
> >
> >> I agree with Itamar -- it feels environmental. I'll do some digging into
> >> the teamcity output but I think the plan of setting up some extra
> verbose
> >> logging here would make sense. I can set you up with a separate build
> >> pointed at your fork if that helps -- it will keep the feedback cycle
> >> tighter. The other thing we could do is categorize the tests and focus
> that
> >> build at running only that category so you don't need to wait on the
> whole
> >> suite to get responses. Let me know if you want me to proceed there.
> >>
> >>
> >>
> >> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <itamar@code972.com
> >
> >> wrote:
> >>
> >>> Yes, that would be the best way to do this. On Java Lucene, the
> >>> randomized
> >>> tests framework allows you to re-use the random seed associated with
> the
> >>> failure, but we are not there yet. Either way, I suspect this to be an
> >>> environment issue rather than a code path one.
> >>>
> >>> --
> >>>
> >>> Itamar Syn-Hershko
> >>> http://code972.com | @synhershko <https://twitter.com/synhershko>
> >>> Freelance Developer & Consultant
> >>> Lucene.NET committer and PMC member
> >>>
> >>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <la...@gmail.com>
> >>> wrote:
> >>>
> >>> > There are three tests that consistently fail on TC but no matter how
> >>> many
> >>> > times I try, I can't reproduce it locally. These tests are:
> >>> >
> >>> > TestFuzzyQuery.TestTieBreaker
> >>> >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> >>> >
> >>> > TestSimpleExplanations.TestDMQ8
> >>> >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> >>> >
> >>> > TestTopDocsMerge.TestSort_2
> >>> >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> >>> >
> >>> > I would fix them if I could reproduce it -- and I am running out of
> >>> ideas
> >>> > how to do it. Even if I put them in a loop running hundreds of
> times, I
> >>> > can't trigger the failure.
> >>> >
> >>> > Anyone have any ideas how to go about reproducing it? I am thinking
> to
> >>> push
> >>> > very verbose code in a separate branch that logs the input values /
> >>> random
> >>> > values that are used and see what happens. Checking if anyone has any
> >>> other
> >>> > suggestions.
> >>> >
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Laimis
> >>> >
> >>>
> >>
> >>
>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
Sorry about the blank one -- getting used to google inbox here and I
misclicked.
Anyhow, I have a repro or at least a rhyme and reason -- TeamCity is
running in release mode and I think we have difffering behavior there. If
you switch your copy of visual studio to release mode you will get the same
failures we are seeing in TeamCity. Does that help narrow it down a bit?
On Sat, May 16, 2015 at 4:26 PM Wyatt Barnett <wy...@gmail.com>
wrote:
>
>
> On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <wy...@gmail.com>
> wrote:
>
>> I agree with Itamar -- it feels environmental. I'll do some digging into
>> the teamcity output but I think the plan of setting up some extra verbose
>> logging here would make sense. I can set you up with a separate build
>> pointed at your fork if that helps -- it will keep the feedback cycle
>> tighter. The other thing we could do is categorize the tests and focus that
>> build at running only that category so you don't need to wait on the whole
>> suite to get responses. Let me know if you want me to proceed there.
>>
>>
>>
>> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <it...@code972.com>
>> wrote:
>>
>>> Yes, that would be the best way to do this. On Java Lucene, the
>>> randomized
>>> tests framework allows you to re-use the random seed associated with the
>>> failure, but we are not there yet. Either way, I suspect this to be an
>>> environment issue rather than a code path one.
>>>
>>> --
>>>
>>> Itamar Syn-Hershko
>>> http://code972.com | @synhershko <https://twitter.com/synhershko>
>>> Freelance Developer & Consultant
>>> Lucene.NET committer and PMC member
>>>
>>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <la...@gmail.com>
>>> wrote:
>>>
>>> > There are three tests that consistently fail on TC but no matter how
>>> many
>>> > times I try, I can't reproduce it locally. These tests are:
>>> >
>>> > TestFuzzyQuery.TestTieBreaker
>>> >
>>> >
>>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
>>> >
>>> > TestSimpleExplanations.TestDMQ8
>>> >
>>> >
>>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
>>> >
>>> > TestTopDocsMerge.TestSort_2
>>> >
>>> >
>>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
>>> >
>>> > I would fix them if I could reproduce it -- and I am running out of
>>> ideas
>>> > how to do it. Even if I put them in a loop running hundreds of times, I
>>> > can't trigger the failure.
>>> >
>>> > Anyone have any ideas how to go about reproducing it? I am thinking to
>>> push
>>> > very verbose code in a separate branch that logs the input values /
>>> random
>>> > values that are used and see what happens. Checking if anyone has any
>>> other
>>> > suggestions.
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > Laimis
>>> >
>>>
>>
>>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
On Sat, May 16, 2015 at 3:22 PM Wyatt Barnett <wy...@gmail.com>
wrote:
> I agree with Itamar -- it feels environmental. I'll do some digging into
> the teamcity output but I think the plan of setting up some extra verbose
> logging here would make sense. I can set you up with a separate build
> pointed at your fork if that helps -- it will keep the feedback cycle
> tighter. The other thing we could do is categorize the tests and focus that
> build at running only that category so you don't need to wait on the whole
> suite to get responses. Let me know if you want me to proceed there.
>
>
>
> On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <it...@code972.com>
> wrote:
>
>> Yes, that would be the best way to do this. On Java Lucene, the randomized
>> tests framework allows you to re-use the random seed associated with the
>> failure, but we are not there yet. Either way, I suspect this to be an
>> environment issue rather than a code path one.
>>
>> --
>>
>> Itamar Syn-Hershko
>> http://code972.com | @synhershko <https://twitter.com/synhershko>
>> Freelance Developer & Consultant
>> Lucene.NET committer and PMC member
>>
>> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <la...@gmail.com>
>> wrote:
>>
>> > There are three tests that consistently fail on TC but no matter how
>> many
>> > times I try, I can't reproduce it locally. These tests are:
>> >
>> > TestFuzzyQuery.TestTieBreaker
>> >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
>> >
>> > TestSimpleExplanations.TestDMQ8
>> >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
>> >
>> > TestTopDocsMerge.TestSort_2
>> >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
>> >
>> > I would fix them if I could reproduce it -- and I am running out of
>> ideas
>> > how to do it. Even if I put them in a loop running hundreds of times, I
>> > can't trigger the failure.
>> >
>> > Anyone have any ideas how to go about reproducing it? I am thinking to
>> push
>> > very verbose code in a separate branch that logs the input values /
>> random
>> > values that are used and see what happens. Checking if anyone has any
>> other
>> > suggestions.
>> >
>> >
>> > Thanks,
>> >
>> > Laimis
>> >
>>
>
>
Re: Reproducing random test failures
Posted by Wyatt Barnett <wy...@gmail.com>.
I agree with Itamar -- it feels environmental. I'll do some digging into
the teamcity output but I think the plan of setting up some extra verbose
logging here would make sense. I can set you up with a separate build
pointed at your fork if that helps -- it will keep the feedback cycle
tighter. The other thing we could do is categorize the tests and focus that
build at running only that category so you don't need to wait on the whole
suite to get responses. Let me know if you want me to proceed there.
On Sat, May 16, 2015 at 3:10 PM, Itamar Syn-Hershko <it...@code972.com>
wrote:
> Yes, that would be the best way to do this. On Java Lucene, the randomized
> tests framework allows you to re-use the random seed associated with the
> failure, but we are not there yet. Either way, I suspect this to be an
> environment issue rather than a code path one.
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko <https://twitter.com/synhershko>
> Freelance Developer & Consultant
> Lucene.NET committer and PMC member
>
> On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <la...@gmail.com>
> wrote:
>
> > There are three tests that consistently fail on TC but no matter how many
> > times I try, I can't reproduce it locally. These tests are:
> >
> > TestFuzzyQuery.TestTieBreaker
> >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> >
> > TestSimpleExplanations.TestDMQ8
> >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> >
> > TestTopDocsMerge.TestSort_2
> >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> >
> > I would fix them if I could reproduce it -- and I am running out of ideas
> > how to do it. Even if I put them in a loop running hundreds of times, I
> > can't trigger the failure.
> >
> > Anyone have any ideas how to go about reproducing it? I am thinking to
> push
> > very verbose code in a separate branch that logs the input values /
> random
> > values that are used and see what happens. Checking if anyone has any
> other
> > suggestions.
> >
> >
> > Thanks,
> >
> > Laimis
> >
>
Re: Reproducing random test failures
Posted by Itamar Syn-Hershko <it...@code972.com>.
Yes, that would be the best way to do this. On Java Lucene, the randomized
tests framework allows you to re-use the random seed associated with the
failure, but we are not there yet. Either way, I suspect this to be an
environment issue rather than a code path one.
--
Itamar Syn-Hershko
http://code972.com | @synhershko <https://twitter.com/synhershko>
Freelance Developer & Consultant
Lucene.NET committer and PMC member
On Sat, May 16, 2015 at 10:06 PM, Laimonas Simutis <la...@gmail.com> wrote:
> There are three tests that consistently fail on TC but no matter how many
> times I try, I can't reproduce it locally. These tests are:
>
> TestFuzzyQuery.TestTieBreaker
>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
>
> TestSimpleExplanations.TestDMQ8
>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
>
> TestTopDocsMerge.TestSort_2
>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
>
> I would fix them if I could reproduce it -- and I am running out of ideas
> how to do it. Even if I put them in a loop running hundreds of times, I
> can't trigger the failure.
>
> Anyone have any ideas how to go about reproducing it? I am thinking to push
> very verbose code in a separate branch that logs the input values / random
> values that are used and see what happens. Checking if anyone has any other
> suggestions.
>
>
> Thanks,
>
> Laimis
>