You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by Wyatt Barnett <wy...@gmail.com> on 2016/12/24 03:12:09 UTC
Hanging TC Builds
Hi guys -- just to let everyone know I'm aware that the builds seem to be
hanging on the CI server. I'm working on figuring out which way is up. From
what I can tell it seems like something locks up and the tests start timing
out until the entire stack melts down.
Will update when I have updates. Happy holidays!
Re: Hanging TC Builds
Posted by Wyatt Barnett <wy...@gmail.com>.
I've been on the road for the last few weeks so I haven't had time to look
into this much. I've got consecutive 3 day weekends coming up so I should
be able to do some tire kicking shortly.
That said, JetBrains did get back to me:
Hi, I tried to rerun successfull build #765
<https://teamcity-support.jetbrains.com/hc/requests/765> (https://teamcity
.jetbrains.com/viewLog.html?buildId=950000&buildTypeId=LuceneNet_LuceneNetNightly&tab=buildResultsDiv)
and go the execution exception (https://teamcity
.jetbrains.com/viewLog.html?buildId=956627&buildTypeId=LuceneNet_LuceneNetNightly&tab=buildResultsDiv).
Comparing parameters gave nothing to me, but after comparing logs I found
this:
[16:31:32] : [Step 3/5] 2016-12-28 13:31:32,116 [824] ERROR JetBrains.
TeamCity.NUnitLauncher.ErrorLoggerImpl - Unhandled exception in remote
appdomain: System.Exception: Expected: 1
[16:31:32] : [Step 3/5] But was: 0
[16:31:32] : [Step 3/5] ---> NUnit.Framework.AssertionException: Expected: 1
[16:31:32] : [Step 3/5] But was: 0
[16:31:32] : [Step 3/5]
[16:31:32] : [Step 3/5] at NUnit.Framework.Assert.That(Object actual,
IResolveConstraint expression, String message, Object[] args)
[16:31:32] : [Step 3/5] at NUnit.Framework.Assert.AreEqual(Int32 expected,
Int32 actual)
[16:31:32] : [Step 3/5] at
Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocument(Term
id, IEnumerable`1 doc) in
C:\BuildAgent\work\5f08f6c5ab8fbc83\src\Lucene.Net.Tests\core\Search\TestControlledRealTimeReopenThread.cs:line
219
[16:31:32] : [Step 3/5] at
Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run()
in
After that point we started ramming into the timeout repeatedly.
My suspicion is a test fails and manages to lock up the disk in a way that
kills performance, making subsequent tests timeout and eventually leading
to the whole suite timing out. One thing to keep in mind is that cloud
stuff is comparable to your desktop in most cases except disk I/O where,
unless one is getting very expensive dedicated SSDs, your PC is probably
orders of magnitude better.
Anyhow, as a way forward I think it might help if someone could walk
through the listed lines above and see if something jumps out at you. I'm
going to try and stand something up at home to simulate poor cloud disk i/o
and see if I can deterministically recreate the slowdowns as well.
Happy new year!
On Mon, Jan 2, 2017 at 7:42 AM Shad Storhaug <sh...@shadstorhaug.com> wrote:
> FYI, I am having similar issues on my branch of #191. The following tests
> are causing NUnit to either crash or not finish. Perhaps these same tests
> are causing issues in master. Excluding them from running during the build
> might be a temporary workaround.
>
> - Lucene.Net.Analysis.Th.TestThaiAnalyzer.TestRandomHugeStrings
> - Lucene.Net.Index.TestByteSlices
> - Lucene.Net.Search.TestSearchAfter.TestQueries
> - Lucene.Net.Util.Fst.TestFSTs.TestRandomWords
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
> -----Original Message-----
> From: itamar.synhershko@gmail.com [mailto:itamar.synhershko@gmail.com] On
> Behalf Of Itamar Syn-Hershko
> Sent: Thursday, December 29, 2016 11:47 PM
> To: dev@lucenenet.apache.org
> Subject: Re: Hanging TC Builds
>
> Hopefully it's the tests code that locks up and not the core - waiting to
> hear news.. Thanks!
>
> --
>
> Itamar Syn-Hershko
> Freelance Developer & Consultant
> Elasticsearch Consulting Partner
> Microsoft MVP | Lucene.NET PMC
> http://code972.com | @synhershko <https://twitter.com/synhershko>
> http://BigDataBoutique.co.il/
>
> On Sat, Dec 24, 2016 at 5:12 AM, Wyatt Barnett <wy...@gmail.com>
> wrote:
>
> > Hi guys -- just to let everyone know I'm aware that the builds seem to
> > be hanging on the CI server. I'm working on figuring out which way is
> > up. From what I can tell it seems like something locks up and the
> > tests start timing out until the entire stack melts down.
> >
> > Will update when I have updates. Happy holidays!
> >
>
RE: Hanging TC Builds
Posted by Shad Storhaug <sh...@shadstorhaug.com>.
FYI, I am having similar issues on my branch of #191. The following tests are causing NUnit to either crash or not finish. Perhaps these same tests are causing issues in master. Excluding them from running during the build might be a temporary workaround.
- Lucene.Net.Analysis.Th.TestThaiAnalyzer.TestRandomHugeStrings
- Lucene.Net.Index.TestByteSlices
- Lucene.Net.Search.TestSearchAfter.TestQueries
- Lucene.Net.Util.Fst.TestFSTs.TestRandomWords
Thanks,
Shad Storhaug (NightOwl888)
-----Original Message-----
From: itamar.synhershko@gmail.com [mailto:itamar.synhershko@gmail.com] On Behalf Of Itamar Syn-Hershko
Sent: Thursday, December 29, 2016 11:47 PM
To: dev@lucenenet.apache.org
Subject: Re: Hanging TC Builds
Hopefully it's the tests code that locks up and not the core - waiting to hear news.. Thanks!
--
Itamar Syn-Hershko
Freelance Developer & Consultant
Elasticsearch Consulting Partner
Microsoft MVP | Lucene.NET PMC
http://code972.com | @synhershko <https://twitter.com/synhershko> http://BigDataBoutique.co.il/
On Sat, Dec 24, 2016 at 5:12 AM, Wyatt Barnett <wy...@gmail.com>
wrote:
> Hi guys -- just to let everyone know I'm aware that the builds seem to
> be hanging on the CI server. I'm working on figuring out which way is
> up. From what I can tell it seems like something locks up and the
> tests start timing out until the entire stack melts down.
>
> Will update when I have updates. Happy holidays!
>
Re: Hanging TC Builds
Posted by Itamar Syn-Hershko <it...@code972.com>.
Hopefully it's the tests code that locks up and not the core - waiting to
hear news.. Thanks!
--
Itamar Syn-Hershko
Freelance Developer & Consultant
Elasticsearch Consulting Partner
Microsoft MVP | Lucene.NET PMC
http://code972.com | @synhershko <https://twitter.com/synhershko>
http://BigDataBoutique.co.il/
On Sat, Dec 24, 2016 at 5:12 AM, Wyatt Barnett <wy...@gmail.com>
wrote:
> Hi guys -- just to let everyone know I'm aware that the builds seem to be
> hanging on the CI server. I'm working on figuring out which way is up. From
> what I can tell it seems like something locks up and the tests start timing
> out until the entire stack melts down.
>
> Will update when I have updates. Happy holidays!
>