You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Etienne Chauchot (Jira)" <ji...@apache.org> on 2021/11/04 10:13:00 UTC
[jira] [Comment Edited] (BEAM-5172)
org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky
[ https://issues.apache.org/jira/browse/BEAM-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17435498#comment-17435498 ]
Etienne Chauchot edited comment on BEAM-5172 at 11/4/21, 10:12 AM:
-------------------------------------------------------------------
Took a look at the 3 flakiness:
1. [https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/19333/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testSplit/] : root cause is that there is no splits on ES6. The only possible cause is estimated index size is < desiredBundleSizeBytes (2000 in the test)
2. [https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4147/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testReadWithQueryString/ |https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4189/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testSizes/]once again the guilty one is the index size estimation estimation is 209 whereas average size is 25 and 40 docs.
=> I recall size estimation is statistical in ES and might vary over time hence the error margin included in the tests. I think we should give the tests more margin but also fix size estimation. I opened this ticket for that. I also opened a cleaning ticket while reviewing the split code.
3. [https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4147/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testReadWithQueryString/ |https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4147/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testReadWithQueryString/]this one worries me more: if there is no scientist matching Einstein out of 40 docs inserted (with 10 different scientists) it is either that the insert was not done or indexed (too much load and docs not yet inserted or not enough wait before indexing time) or the match query was not properly passed by the queryconfigurer in the unit test.
[~egalpin] can you take a look with this hints in mind ?
was (Author: echauchot):
Took a look at the 3 flakiness:
1. [https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/19333/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testSplit/] : root cause is that there is no splits on ES6. The only possible cause is estimated index size is < desiredBundleSizeBytes (2000 in the test)
2. [https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4147/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testReadWithQueryString/ |https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4189/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testSizes/]once again the guilty one is the index size estimation estimation is 209 whereas average size is 25 and 40 docs.
=> I recall size estimation is statistical in ES and might vary over time hence the error margin included in the tests. I think we should give the tests more margin but also fix size estimation. I opened [this|https://issues.apache.org/jira/browse/BEAM-13137] ticket for that. I also opened a [cleaning ticket|https://issues.apache.org/jira/browse/BEAM-13136] while reviewing the split code.
3. [https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4147/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testReadWithQueryString/ |https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4147/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testReadWithQueryString/]this once worries me more: if there is no scientist matching Einstein out of 40 docs inserted (with 10 different scientists) it is either that the insert was not done or indexed (too much load and docs not yet inserted or not enough wait before indexing time) or the match query was not properly passed by the queryconfigurer in the unit test.
[~egalpin] can you take a look with this hints in mind ?
> org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky
> -----------------------------------------------------------------
>
> Key: BEAM-5172
> URL: https://issues.apache.org/jira/browse/BEAM-5172
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch, test-failures
> Affects Versions: 2.31.0
> Reporter: Valentyn Tymofieiev
> Priority: P1
> Labels: flake, sickbay
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> In a recent PostCommit builld, https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1290/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testRead/ failed with:
> Error Message
> java.lang.AssertionError: Count/Flatten.PCollections.out:
> Expected: <400L>
> but: was <470L>
> Stacktrace
> java.lang.AssertionError: Count/Flatten.PCollections.out:
> Expected: <400L>
> but: was <470L>
> at org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:168)
> at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:413)
> at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:404)
> at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestCommon.testRead(ElasticsearchIOTestCommon.java:124)
> at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testRead(ElasticsearchIOTest.java:125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
> at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
> at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
> at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
> Caused by: java.lang.AssertionError:
> Expected: <400L>
> but: was <470L>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.junit.Assert.assertThat(Assert.java:923)
> at org.apache.beam.sdk.testing.PAssert$AssertIsEqualTo.apply(PAssert.java:1264)
> at org.apache.beam.sdk.testing.PAssert$AssertIsEqualTo.apply(PAssert.java:1254)
> at org.apache.beam.sdk.testing.PAssert$CheckRelationAgainstExpected.apply(PAssert.java:960)
> at org.apache.beam.sdk.testing.PAssert$CheckRelationAgainstExpected.apply(PAssert.java:940)
> at org.apache.beam.sdk.testing.PAssert.doChecks(PAssert.java:1241)
> at org.apache.beam.sdk.testing.PAssert$SideInputCheckerDoFn.processElement(PAssert.java:1185)
> at org.apache.beam.sdk.testing.PAssert$SideInputCheckerDoFn$DoFnInvoker.invokeProcessElement(Unknown Source)
> at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:275)
> at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:237)
> at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:87)
> at org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:207)
> at org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:55)
> at org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
> at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Standard Error
--
This message was sent by Atlassian Jira
(v8.3.4#803005)