You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Rui Li (JIRA)" <ji...@apache.org> on 2014/09/25 05:45:34 UTC
[jira] [Commented] (HIVE-7382) Create a MiniSparkCluster and set up
a testing framework [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147328#comment-14147328 ]
Rui Li commented on HIVE-7382:
------------------------------
Hi [~xuefuz],
Just to clarify, we currently use spark local mode to run the tests. And our goal here is to use local-cluster mode to run the test, right?
So far I've found, local mode runs all the spark executor, backend, etc. in the same JVM, while in local-cluster mode, master and workers run in the same JVM and the executors run in separate JVMs. Local-cluster mode resembles the standalone mode, except that master and workers run in the same JVM and everything runs on a single machine.
Intuitively, cluster-local mode can catch more errors we may have. But this mode seems to be intended only for spark tests and not exposed to users. We may need to make sure if we really want to use it.
> Create a MiniSparkCluster and set up a testing framework [Spark Branch]
> -----------------------------------------------------------------------
>
> Key: HIVE-7382
> URL: https://issues.apache.org/jira/browse/HIVE-7382
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: Rui Li
> Labels: Spark-M1
>
> To automatically test Hive functionality over Spark execution engine, we need to create a test framework that can execute Hive queries with Spark as the backend. For that, we should create a MiniSparkCluser for this, similar to other execution engines.
> Spark has a way to create a local cluster with a few processes in the local machine, each process is a work node. It's fairly close to a real Spark cluster. Our mini cluster can be based on that.
> For more info, please refer to the design doc on wiki.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)