spark submit to zeppelin built in Local mode

[pr...@wipro.com]

Hi,

Is it possible to do a spark_submit with zeppelin built in Local mode ? The Notebook accepts only code snippets. So that is not an alternative. Eg: The Notebook does not allow package names. A line like: package com.mycompany.settlement  results in a parse error in the Notebook.

If it is possible to submit existing code to the Local mode's spark interpreter, that would be very useful for developers, by avoiding the need to run zeppelin and spark as separate servers.

Thanks,
Prajod
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com

RE: spark submit to zeppelin built in Local mode

[pr...@wipro.com]

Hi,

I was actually trying to integrate Flume->Spark->Zeppelin. I used the %dep dependency loader to load the compiled packages, like you mentioned. Then I invoked the main class from my package by defining an instance variable on the REPL using %spark interpreter. It worked partially proving that such invocation is possible.

But in my use case, Flume needs the port number of the Spark master. Since the REPL does not use a port, I got an exception (connect exception).

Regards,
Prajod

From: moon soo Lee [mailto:moon@apache.org]
Sent: 28 April 2015 04:55
To: users@zeppelin.incubator.apache.org
Subject: Re: spark submit to zeppelin built in Local mode

Hi Prajod,

Zeppelin uses scala REPL in it's SparkInterpreter and scala REPL not allowing defining a package.
To submit existing code, http://zeppelin.incubator.apache.org/docs/interpreter/spark.html#dependencyloading might help. Is it helping you?

Thanks,
moon


On Tue, Apr 28, 2015 at 2:34 AM <pr...@wipro.com>> wrote:
Hi,

Is it possible to do a spark_submit with zeppelin built in Local mode ? The Notebook accepts only code snippets. So that is not an alternative. Eg: The Notebook does not allow package names. A line like: package com.mycompany.settlement  results in a parse error in the Notebook.

If it is possible to submit existing code to the Local mode’s spark interpreter, that would be very useful for developers, by avoiding the need to run zeppelin and spark as separate servers.

Thanks,
Prajod
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com<http://www.wipro.com>
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com

Re: spark submit to zeppelin built in Local mode

[moon soo Lee <mo...@apache.org>]

Hi Prajod,

Zeppelin uses scala REPL in it's SparkInterpreter and scala REPL not
allowing defining a package.
To submit existing code,
http://zeppelin.incubator.apache.org/docs/interpreter/spark.html#dependencyloading
might
help. Is it helping you?

Thanks,
moon



On Tue, Apr 28, 2015 at 2:34 AM <pr...@wipro.com> wrote:

>  Hi,
>
>
>
> Is it possible to do a spark_submit with zeppelin built in Local mode ?
> The Notebook accepts only code snippets. So that is not an alternative. Eg:
> The Notebook does not allow package names. A line like: package
> com.mycompany.settlement  results in a parse error in the Notebook.
>
>
>
> If it is possible to submit existing code to the Local mode’s spark
> interpreter, that would be very useful for developers, by avoiding the need
> to run zeppelin and spark as separate servers.
>
>
>
> Thanks,
>
> Prajod
>  The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments. WARNING: Computer viruses can be
> transmitted via email. The recipient should check this email and any
> attachments for the presence of viruses. The company accepts no liability
> for any damage caused by any virus transmitted by this email.
> www.wipro.com
>