[jira] [Created] (SPARK-3691) Provide a mini cluster for testing system built on Spark

["Xuefu Zhang (JIRA)" <ji...@apache.org>]

Xuefu Zhang created SPARK-3691:
----------------------------------

             Summary: Provide a mini cluster for testing system built on Spark
                 Key: SPARK-3691
                 URL: https://issues.apache.org/jira/browse/SPARK-3691
             Project: Spark
          Issue Type: Test
          Components: Spark Core
    Affects Versions: 1.1.0
            Reporter: Xuefu Zhang


Most Hadoop components such MR, DFS, Tez, and Yarn provide a mini cluster that can be used to test the external systems that rely on those frameworks, such as Pig and Hive. While Spark's local mode can be used to do such testing and is friendly for debugging, it's too far from a real Spark cluster and a lot of problems cannot be discovered. Thus, an equivalent of Hadoop MR mini cluster in Spark would be very helpful in testing system such as Hive/Pig on Spark.

Spark's local-cluster is considered for this purpose but it doesn't fit well because it requires a Spark installation on the box where the tests run. Also, local-cluster isn't exposed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org