You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2014/12/19 17:14:13 UTC
[jira] [Updated] (SPARK-3912) FlumeStreamSuite is flaky, fails
either with port binding issues or data not being reliably sent
[ https://issues.apache.org/jira/browse/SPARK-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen updated SPARK-3912:
------------------------------
Labels: flaky-test (was: )
> FlumeStreamSuite is flaky, fails either with port binding issues or data not being reliably sent
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-3912
> URL: https://issues.apache.org/jira/browse/SPARK-3912
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.1.0
> Reporter: Tathagata Das
> Assignee: Tathagata Das
> Priority: Critical
> Labels: flaky-test
> Fix For: 1.2.0
>
>
> Two problems.
> 1. Attempts to start the service to start on different possible ports (to avoid bind failures) was incorrect as the service is actually start lazily (when receiver starts, not when the flume input stream is created).
> 2. Lots of Thread.sleep was used to improve the probabilities that data sent through avro to flume receiver was being sent. However, the sending may fail for various unknown reasons, causing the test to fail.
> 3. Thread.sleep was also used to send one record per batch and checks were made on whether only one records was received in every batch. This was an overkill because all we need to test in this unit test is whether data is being sent and received or not, not about timings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org