[jira] [Commented] (BEAM-5813) ElasticSearchIOTest.testSplit is flaky

["Etienne Chauchot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686801#comment-16686801 ] 

Etienne Chauchot commented on BEAM-5813:
----------------------------------------

I submitted a PR that might correct that. Copying my assumption from the PR:
From time to time, the expected number of splits is higher that the actual number of sources after the split. But both expected and actual number of splits are determined using {{Math.ceil(indexSize/desiredBundleSize)}}. My guess is that the difference is due to indexSize not being the same between the time when expected value is calculated and the time when it is actually estimated in split. This can have 2 reasons:
1- index size in ES has a part of random (says Elastic dev team)
2- refresh of the index was not finished when split is done.
=> This PR addresses point 2 because there is nothing we can do about point 1: test util requests to ES are issued *synchronously* and they specify refresh=true which causes a refresh to be done but I'm not sure that this refresh is synchronous. Instead use refresh=wait so that the request waits for the index refresh before returning.

We need to wait some time (1 week?) to see the results of the postCommits to see if there is still flakiness on the overloaded jenkins (I cannot reproduce it on my machine).

> ElasticSearchIOTest.testSplit is flaky
> --------------------------------------
>
>                 Key: BEAM-5813
>                 URL: https://issues.apache.org/jira/browse/BEAM-5813
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-elasticsearch
>            Reporter: Kenneth Knowles
>            Assignee: Etienne Chauchot
>            Priority: Critical
>              Labels: flake
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1724/]
> [https://scans.gradle.com/s/xpsswtkdf62so/tests/y3up3nrnbr2ns-opjrlgkblx2ku]
> {code:java}
> Wrong number of splits expected:<25> but was:<22>
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)