Jenkins build is back to normal : Mahout-Quality #2166

[Apache Jenkins Server <je...@builds.apache.org>]

See <https://builds.apache.org/job/Mahout-Quality/2166/>

Re: Jenkins build is back to normal : Mahout-Quality #2166

[Stevo Slavić <ss...@gmail.com>]

Mahout build is oscillating because test SearchSanityTest.testRemoval is
oscillating. When it fails, it fails for LocalitySensitiveHashSearch
searcher only.

I've run this test couple of times in IDE, most of the time it's
successful, but occasionally it fails.
With breakpoint on line 166, and breakpoint condition
Math.abs(r.get(0).getValue().minus(r0.get(1).getValue()).norm (1) - 0) >
1.0e-8

one of the failures had following variable state:

x =
[{0:0.8709558437987425,1:-0.5375954118626808,2:-1.2425196954788476,3:-0.5963352681084106,4:1.0979837347756136,5:2.304185373890536,6:0.25793628051644957,7:-1.134330442919324,8:1.597646398122005,9:0.8298520595445593,10:0.6261553451858568,11:-1.1887039545363631,12:0.5869327492264573,13:0.28388860533192856,14:-1.3022503895547264,15:0.6010871637821463,16:0.746145422813361,17:1.1176489418882314,18:0.12290804380398342,19:-0.7023943607753743},
{0:-0.1273613082447137,1:0.5212805226355095,2:-1.3441436944174596,3:0.726170066826851,4:0.771368418859047,5:0.01430669137394401,6:-1.0322788220328099,7:0.9800796113367815,8:1.0860923051790574,9:-0.5343931502079539,10:-0.8687519451795613,11:-0.4882818867652608,12:0.5763217636362794,13:-1.5319064044424902,14:1.5805638933745385,15:1.1042306701927427,16:0.8039127800857652,17:0.06535662950296073,18:0.16616159580914874,19:0.5477575460433336}]

r0 =
[0]=>{0:0.8709558437987425,1:-0.5375954118626808,2:-1.2425196954788476,3:-0.5963352681084106,4:1.0979837347756136,5:2.304185373890536,6:0.25793628051644957,7:-1.134330442919324,8:1.597646398122005,9:0.8298520595445593,10:0.6261553451858568,11:-1.1887039545363631,12:0.5869327492264573,13:0.28388860533192856,14:-1.3022503895547264,15:0.6010871637821463,16:0.746145422813361,17:1.1176489418882314,18:0.12290804380398342,19:-0.7023943607753743}
weight 0

[1]=>
{0:-0.20483940907349352,1:0.11976413233109141,2:-0.446414355106293,3:-1.7367952330678786,4:0.38238934234736355,5:0.6975417327142289,6:0.3373969986394851,7:-1.4185380951449957,8:0.9265441416353332,9:-0.025206163271980428,10:0.4669637747091974,11:-0.5737988084119644,12:0.41358433275511775,13:0.177386844272387,14:-1.346843287707939,15:0.3409985498083112,16:0.273396568351321,17:0.535280502127145,18:-0.903059026848925,19:-0.8379568998394694}
weight 3.1676437428926834

[2]=>
{0:0.8249102943426255,1:-1.3029973079099118,2:-0.6637757770704698,3:-0.6872922698053289,4:0.8694850261953001,5:2.0098551642810314,6:0.6329036548247581,7:-0.7020002439436349,8:1.1002127917459246,9:0.9940539101040394,10:1.2930352335192057,11:1.2969741235560996,12:0.5655788109525552,13:0.3352483698563293,14:-0.07778210967771995,15:0.21446631032850147,16:0.19873363445983003,17:-0.299301008269592,18:-0.3475216337380752,19:-0.6413611008146741}
weight 3.5326455510724935

[3]=>
{0:1.0213160889001143,1:-0.16764174017854278,2:-0.8248105102044536,3:0.548181150451324,4:1.2389184164722173,5:0.2894603912982171,6:0.4878734279209343,7:-0.880556725985863,8:1.826747666612512,9:-0.4546221613342545,10:0.3506596412191718,11:-0.6414730833164245,12:-0.13282164842381447,13:0.45057016018778245,14:-1.5111928535925796,15:0.06638106718301788,16:0.08281114228255751,17:-0.6421195739702901,18:-0.5493966162085773,19:-0.007304932871986156}
weight 3.642139801900397
...

r =
[0]=>
{0:0.8249102943426255,1:-1.3029973079099118,2:-0.6637757770704698,3:-0.6872922698053289,4:0.8694850261953001,5:2.0098551642810314,6:0.6329036548247581,7:-0.7020002439436349,8:1.1002127917459246,9:0.9940539101040394,10:1.2930352335192057,11:1.2969741235560996,12:0.5655788109525552,13:0.3352483698563293,14:-0.07778210967771995,15:0.21446631032850147,16:0.19873363445983003,17:-0.299301008269592,18:-0.3475216337380752,19:-0.6413611008146741}
weight 3.5326455510724935

[1]=>
{0:1.0213160889001143,1:-0.16764174017854278,2:-0.8248105102044536,3:0.548181150451324,4:1.2389184164722173,5:0.2894603912982171,6:0.4878734279209343,7:-0.880556725985863,8:1.826747666612512,9:-0.4546221613342545,10:0.3506596412191718,11:-0.6414730833164245,12:-0.13282164842381447,13:0.45057016018778245,14:-1.5111928535925796,15:0.06638106718301788,16:0.08281114228255751,17:-0.6421195739702901,18:-0.5493966162085773,19:-0.007304932871986156}
weight 3.642139801900397

[2]=>
{0:1.6656465845840556,1:0.6717244738768022,2:-0.11589200853467475,3:-1.5058325112372222,4:-0.6229838627352531,5:0.9744547121420478,6:0.5863699160394856,7:-1.2460776338162076,8:1.7748095928030267,9:0.7468726003405439,10:0.6084109063104702,11:-1.8578009892923697,12:-0.8413781857292507,13:0.6656257410413803,14:-0.6330806168110932,15:0.095425316213816,16:1.208742880515909,17:0.6308515482408196,18:0.8135753619367572,19:0.2227766502790239}
weight 3.7683177596914277

[3]=>
{0:1.0178665512250946,1:-0.6883855950038557,2:-0.04178788478187503,3:-0.538402076928077,4:0.6054117921440514,5:0.3977226709629301,6:-0.9347093442045187,7:-0.30351297076438905,8:1.443995518669909,9:0.5206899006395286,10:0.6910907561625238,11:-0.9735926881509751,12:0.03088350548161232,13:1.9198317061992651,14:-0.023091788306616047,15:-0.646778674755111,16:1.3667190445788564,17:0.948728984611376,18:-0.7003219473537609,19:-0.07116120942226048}
weight 3.9137866873451608

[4]=>
{0:-0.06104396089698967,1:0.3582467872592491,2:-1.2321617926950135,3:-0.3063964801427399,4:0.5705561219335118,5:1.635159655563086,6:-0.4322617547074639,7:0.35200968498141033,8:1.1255343874841222,9:1.2954853545350034,10:-1.2433021691017543,11:-0.38815748993665855,12:-0.32746700116554345,13:0.8518245887139779,14:-0.6809745908671637,15:1.2771441177793372,16:-0.8748542965138818,17:0.7189500954859568,18:0.2863596417106317,19:-1.6684649951412196}
weight 3.9357920409937517
...

First I noticed that both r0 and r have 30 elements each, which is searcher
search size in test. If I'm not mistaken
"LocalitySensitiveHashSearch.search(Vector query, int limit)" should
instead return up to a "limit" of results, but there is currently no logic
for limiting results returned, apart from searcher's search size. Attached
is the LocalitySensitiveHashSearch.java.patch which fixes this, please
review it.


On with actual issue - as values in r show, second search in the test
doesn't find r0[1] to be new nearest neighbor. Searcher size reports that
only single vector has been removed, as trainingVectors has size of 8191.

To check if that r0[1] vector is still in, I've evaluated expression:
searcher.search(r0.get(1).getValue(), 1)

and vector was found by the searcher,
{0:-0.20483940907349352,1:0.11976413233109141,2:-0.446414355106293,3:-1.7367952330678786,4:0.38238934234736355,5:0.6975417327142289,6:0.3373969986394851,7:-1.4185380951449957,8:0.9265441416353332,9:-0.025206163271980428,10:0.4669637747091974,11:-0.5737988084119644,12:0.41358433275511775,13:0.177386844272387,14:-1.346843287707939,15:0.3409985498083112,16:0.273396568351321,17:0.535280502127145,18:-0.903059026848925,19:-0.8379568998394694}
weight 0.0


On another occasion when this test failed for me in IDE, r[0] weight was
smaller than weight of r0[1].

For a moment I thought it had to do with double comparison in
Searcher.getCandidateQueue, anonymous PriorityQueue and its lessThan
implementation (see Searcher.java.patch), but that didn't fix the issue in
searcher, still could reproduce failing test.

So, there is truly a bug in LocalitySensitiveHashSearch searcher - after
removal of a nearest neighbor vector it can happen that searcher will not
return same r0[1] vector in second search results, to be new nearest
neighbor of x[0].

Kind regards,
Stevo Slavic.



On Mon, Jul 29, 2013 at 12:40 AM, Apache Jenkins Server <
jenkins@builds.apache.org> wrote:

> See <https://builds.apache.org/job/Mahout-Quality/2166/>
>
>