Re: Integrate Spark Editor with Hue for source compiled installation of spark/spark-jobServer

[Sunita Arvind <su...@gmail.com>]

That's good to know. I will try it out.

Thanks Romain

On Friday, June 27, 2014, Romain Rigaux <ro...@gmail.com> wrote:

> So far Spark Job Server does not work with Spark 1.0:
> https://github.com/ooyala/spark-jobserver
>
> So this works only with Spark 0.9 currently:
>
> http://gethue.com/get-started-with-spark-deploy-spark-server-and-compute-pi-from-your-web-browser/
>
> Romain
>
>
>
> Romain
>
>
> On Tue, Jun 24, 2014 at 9:04 AM, Sunita Arvind <sunitarvind@gmail.com
> <javascript:_e(%7B%7D,'cvml','sunitarvind@gmail.com');>> wrote:
>
>> Hello Experts,
>>
>> I am attempting to integrate Spark Editor with Hue on CDH5.0.1. I have
>> the spark installation build manually from the sources for spark1.0.0. I am
>> able to integrate this with cloudera manager.
>>
>> Background:
>> -------------------
>> We have a 3 node VM cluster with CDH5.0.1
>> We requried spark1.0.0 due to some features in it, so I did a
>>
>>  "yum remove spark-core spark-master spark-worker spark-python"
>>
>>  of the default spark0.9.0 and compiled spark1.0.0 from source:
>>
>> Downloaded the spark-trunk from
>>
>> git clone https://github.com/apache/spark.git
>> cd spark
>> SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true ./sbt/sbt assembly
>>
>>  The spark-assembly-1.0.0-SNAPSHOT-hadoop2.2.0.jar was built and spark by
>> itself seems to work well. I was even able to run a text file count.
>>
>> Current attempt:
>> ----------------------------
>> Referring to this article -
>> http://gethue.com/a-new-spark-web-ui-spark-app/
>> Now I am trying to add the Spark editor to Hue. AFAIK, this requires
>> git clone https://github.com/ooyala/spark-jobserver.git
>> cd spark-jobserver
>> sbt
>> re-start
>>
>> This was successful after lot of struggle with the proxy settings.
>> However, is this the job Server itself? Will that mean the job Server has
>> to be manually started. I intend to have the spark editor show up in hue
>> web UI and I am no way close. Can some one please help?
>>
>> Note, the 3 VMs are Linux CentOS. Not sure if setting something like can
>> be expected to work.:
>>
>> [desktop]
>> app_blacklist=
>>
>>
>> Also, I have made the changes to vim .
>> /job-server/src/main/resources/application.conf as recommended, however,
>> I do not expect this to impact hue in any way.
>>
>> Also, I intend to let the editor stay available, not spawn it everytime
>> it is required.
>>
>>
>> Thanks in advance.
>>
>> regards
>>
>
>