You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/11/14 15:48:01 UTC
[jira] [Work logged] (BEAM-5813) ElasticSearchIOTest.testSplit is
flaky
[ https://issues.apache.org/jira/browse/BEAM-5813?focusedWorklogId=165930&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-165930 ]
ASF GitHub Bot logged work on BEAM-5813:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Nov/18 15:47
Start Date: 14/Nov/18 15:47
Worklog Time Spent: 10m
Work Description: echauchot opened a new pull request #7038: [BEAM-5813] wait for the refresh of ES indices in all test utils that do index writes.
URL: https://github.com/apache/beam/pull/7038
Testsplit is flaky at least on ES6. From time to time, the expected number of splits is higher that the actual number of sources after the split. But both expected and actual number of splits are determined using `Math.ceil(indexSize/desiredBundleSize)`. My guess is that the difference is due to `indexSize` not being the same between the time when expected value is calculated and the time when it is actually estimated in split. This can have 2 reasons:
1- index size in ES has a part of random (says Elastic dev team)
2- refresh of the index was not finished when split is done.
=> This PR addresses point 2 because there is nothing we can do about point 1: test util requests to ES are issued **synchronously** and they specify `refresh=true` which causes a refresh to be done but I'm not sure that this refresh is synchronous. Instead use `refresh=wait` so that the request waits for the index refresh before returning.
R: @timrobertson100
CC: @kennknowles
**Please** add a meaningful description for your change here
------------------------
Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 165930)
Time Spent: 10m
Remaining Estimate: 0h
> ElasticSearchIOTest.testSplit is flaky
> --------------------------------------
>
> Key: BEAM-5813
> URL: https://issues.apache.org/jira/browse/BEAM-5813
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch
> Reporter: Kenneth Knowles
> Assignee: Etienne Chauchot
> Priority: Critical
> Labels: flake
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1724/]
> [https://scans.gradle.com/s/xpsswtkdf62so/tests/y3up3nrnbr2ns-opjrlgkblx2ku]
> {code:java}
> Wrong number of splits expected:<25> but was:<22>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)