Hive on Spark for simple hive queries

["Prabu Soundar Rajan -X (prabsoun - MINDTREE LIMITED at Cisco)" <pr...@cisco.com>]

Hi Team,

In spite of setting hive execution engine as Spark, when we try simple hive queries having only mapper phase like (select * from table where column=xyz) - we observe the jobs are not submitted to Spark master. We do not see those jobs in Spark master web UI. But when we try some queries with reducer phase(in mr execution style), we see the job as "Hive on Spark" application in Spark master web UI. Appreciate if you could help us understand this behavior.   Am I missing something obvious here?

Thanks & Regards,
Prabu

Re: Hive on Spark for simple hive queries

[Xuefu Zhang <xz...@cloudera.com>]

For certain simple queries like your example, Hive doesn't some
optimization by execute it locally, which means no jobs are submitted to
the cluster (MR or Spark). I'm not sure if there is a way to turn this off,
but this is true for all execution engines.

Thanks,
Xuefu

On Tue, Nov 4, 2014 at 8:36 AM, Prabu Soundar Rajan -X (prabsoun - MINDTREE
LIMITED at Cisco) <pr...@cisco.com> wrote:

> Hi Team,
>
> In spite of setting hive execution engine as Spark, when we try simple
> hive queries having only mapper phase like (select * from table where
> column=xyz) - we observe the jobs are not submitted to Spark master. We do
> not see those jobs in Spark master web UI. But when we try some queries
> with reducer phase(in mr execution style), we see the job as "Hive on
> Spark" application in Spark master web UI. Appreciate if you could help us
> understand this behavior.   Am I missing something obvious here?
>
> Thanks & Regards,
> Prabu
>
>