You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/12/24 14:58:49 UTC
[jira] [Updated] (SPARK-10372) Add end-to-end tests for the
scheduling code
[ https://issues.apache.org/jira/browse/SPARK-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-10372:
------------------------------
Target Version/s: (was: 1.6.0)
> Add end-to-end tests for the scheduling code
> --------------------------------------------
>
> Key: SPARK-10372
> URL: https://issues.apache.org/jira/browse/SPARK-10372
> Project: Spark
> Issue Type: Sub-task
> Components: Scheduler, Tests
> Affects Versions: 1.5.0
> Reporter: Imran Rashid
> Assignee: Imran Rashid
>
> The current testing framework for the scheduler only tests individual classes in isolation: {{DAGSchedulerSuite}}, {{TaskSchedulerImplSuite}}, etc. Of course that is useful, but we are missing tests which cover the interaction between these components. We also have larger tests which run entire spark jobs, but that doesn't allow fine grained control of failures for verifying spark's fault-tolerance.
> Adding a framework for testing the scheduler as a whole will:
> 1. Allow testing bugs which involve the interaction between multiple parts of the scheduler, eg. SPARK-10370
> 2. Greater confidence in refactoring the scheduler as a whole. Given the tight coordination between the components its hard to consider any refactoring, since it would be unlikely to be covered by any tests.
> 3. Make it easier to increase test coverage. Writing tests for the {{DAGScheduler}} now requires intimate knowledge of exactly how the components fit together -- a lot of work goes into mimicking the appropriate behavior of the other components. Furthermore, it makes the tests harder to understand for the un-initiated -- which parts are simulating some condition of an external system (eg., losing an executor), and which parts are just interaction with other parts of the scheduler (eg., task resubmission)? These tests will allow to just work at the level of the interaction w/ the executors -- tasks complete, tasks fail, executors are lost, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org