You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Abhi Basu <90...@gmail.com> on 2016/02/17 19:18:45 UTC
Zeppelin 0.56 binaries
Will the binaries work right out of the box with CDH 5.5 and Spark 1.5.0 on
YARN? Or, do I need to build from source as this describes:
https://zeppelin.incubator.apache.org/docs/0.5.5-incubating/install/yarn_install.html
?
Thanks,
Abhi
--
Abhi Basu
Re: Zeppelin 0.56 binaries
Posted by Felix Cheung <fe...@hotmail.com>.
If it is running you should be able to see it on he YARN service Application tab in Cloudera Manager. It should also be on Yarn history server page as well.
On Thu, Feb 18, 2016 at 2:37 PM -0800, "moon soo Lee" <mo...@apache.org> wrote:
'zeppelin-root-cdh-lsedge.local.com.out' file looks like have your Spark
logs. And if your ResourceManager shows Zeppelin in RUNNING application
section after you start Zeppelin and run some code using SparkInterpreter,
you can sure Zeppelin is working on Spark and Yarn.
I don't have any better idea to verify the configuration. If someone have
any other idea please share.
Maybe print hostnames where your code runs?
%spark
import sys.process._
sc.parallelize(1 to 10).map(i=> (("hostname" !!).replaceAll("\n", ""),
1)).reduceByKey(_ + _).collect
Thanks,
moon
On Thu, Feb 18, 2016 at 1:32 PM Abhi Basu <90...@gmail.com> wrote:
> Moon:
>
> Need a clarification. I am running off the binary package off Zeppelin
> 0.56 here on CDH 5.5.1. Spark and Hadoop paths are exported in
> zeppelin-env.sh. Spark and Hive interpreters are configured. Ran a bunch of
> commands from Zeppelin - Hive, PySpark and SparkSQL. Dont see a log for
> Spark interpreter in the $ZEPPELIN_HOME/logs folder. I have also attached
> the yarn app log and that shows Zeppelin is using Yarn. Where can I see the
> Spark logs?
>
> I want to verify Zeppelin is working on Spark and YARN.
>
> Thanks,
>
> Abhi
>
> On Thu, Feb 18, 2016 at 12:09 PM, moon soo Lee <mo...@apache.org> wrote:
>
>> Right, after https://issues.apache.org/jira/browse/ZEPPELIN-160 resolved,
>> Zeppelin supposed to work with any (supported) version of Spark without
>> rebuild. Let us know if it does not work.
>>
>> Binary package is created by dev/create_release.sh script and you can
>> check build flag
>> https://github.com/apache/incubator-zeppelin/blob/v0.5.6/dev/create_release.sh#L128 used
>> for the release.
>>
>> And thanks for pointing that install docs make confusion. I also think
>> they need to be updated.
>>
>> Thanks,
>> moon
>>
>> On Thu, Feb 18, 2016 at 12:01 PM Abhi Basu <90...@gmail.com> wrote:
>>
>>> Ok, thanks for the reply.
>>>
>>> So, from what you are saying, binary package should just use the
>>> installed hadoop and spark versions, but just editing the zeppelin config
>>> files - hadoop conf dir and spark home.
>>>
>>> Out of the box this would also support YARN, without being built with
>>> -pYarn switch, as we had to before? Are the binaries build with that switc
>>> already.
>>>
>>> Updating the install docs for binary would be a great way to lose some
>>> confusion. I have been using Zeppelin for a while and earlier we had to
>>> build from source specifying CDH, Spark versions and various switches.
>>>
>>> Just want to clarify before wasting time building from source :)
>>>
>>> BTW, awesome tool, love it :)
>>>
>>> Thanks,
>>>
>>> Abhi
>>>
>>> On Thu, Feb 18, 2016 at 11:03 AM, moon soo Lee <mo...@apache.org> wrote:
>>>
>>>> Hi Abhi,
>>>>
>>>> Zeppelin 0.5.6-incubating binary package supposed to load installed
>>>> Spark binaries when SPARK_HOME is exported in conf/zeppelin-env.sh.
>>>>
>>>> Please let us know if it does not work with CDH 5.5.
>>>>
>>>> Thanks,
>>>> moon
>>>>
>>>>
>>>> On Wed, Feb 17, 2016 at 10:19 AM Abhi Basu <90...@gmail.com> wrote:
>>>>
>>>>> Will the binaries work right out of the box with CDH 5.5 and Spark
>>>>> 1.5.0 on YARN? Or, do I need to build from source as this describes:
>>>>> https://zeppelin.incubator.apache.org/docs/0.5.5-incubating/install/yarn_install.html
>>>>> ?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Abhi
>>>>>
>>>>> --
>>>>> Abhi Basu
>>>>>
>>>>
>>>
>>>
>>> --
>>> Abhi Basu
>>>
>>
>
>
> --
> Abhi Basu
>
Re: Zeppelin 0.56 binaries
Posted by moon soo Lee <mo...@apache.org>.
'zeppelin-root-cdh-lsedge.local.com.out' file looks like have your Spark
logs. And if your ResourceManager shows Zeppelin in RUNNING application
section after you start Zeppelin and run some code using SparkInterpreter,
you can sure Zeppelin is working on Spark and Yarn.
I don't have any better idea to verify the configuration. If someone have
any other idea please share.
Maybe print hostnames where your code runs?
%spark
import sys.process._
sc.parallelize(1 to 10).map(i=> (("hostname" !!).replaceAll("\n", ""),
1)).reduceByKey(_ + _).collect
Thanks,
moon
On Thu, Feb 18, 2016 at 1:32 PM Abhi Basu <90...@gmail.com> wrote:
> Moon:
>
> Need a clarification. I am running off the binary package off Zeppelin
> 0.56 here on CDH 5.5.1. Spark and Hadoop paths are exported in
> zeppelin-env.sh. Spark and Hive interpreters are configured. Ran a bunch of
> commands from Zeppelin - Hive, PySpark and SparkSQL. Dont see a log for
> Spark interpreter in the $ZEPPELIN_HOME/logs folder. I have also attached
> the yarn app log and that shows Zeppelin is using Yarn. Where can I see the
> Spark logs?
>
> I want to verify Zeppelin is working on Spark and YARN.
>
> Thanks,
>
> Abhi
>
> On Thu, Feb 18, 2016 at 12:09 PM, moon soo Lee <mo...@apache.org> wrote:
>
>> Right, after https://issues.apache.org/jira/browse/ZEPPELIN-160 resolved,
>> Zeppelin supposed to work with any (supported) version of Spark without
>> rebuild. Let us know if it does not work.
>>
>> Binary package is created by dev/create_release.sh script and you can
>> check build flag
>> https://github.com/apache/incubator-zeppelin/blob/v0.5.6/dev/create_release.sh#L128 used
>> for the release.
>>
>> And thanks for pointing that install docs make confusion. I also think
>> they need to be updated.
>>
>> Thanks,
>> moon
>>
>> On Thu, Feb 18, 2016 at 12:01 PM Abhi Basu <90...@gmail.com> wrote:
>>
>>> Ok, thanks for the reply.
>>>
>>> So, from what you are saying, binary package should just use the
>>> installed hadoop and spark versions, but just editing the zeppelin config
>>> files - hadoop conf dir and spark home.
>>>
>>> Out of the box this would also support YARN, without being built with
>>> -pYarn switch, as we had to before? Are the binaries build with that switc
>>> already.
>>>
>>> Updating the install docs for binary would be a great way to lose some
>>> confusion. I have been using Zeppelin for a while and earlier we had to
>>> build from source specifying CDH, Spark versions and various switches.
>>>
>>> Just want to clarify before wasting time building from source :)
>>>
>>> BTW, awesome tool, love it :)
>>>
>>> Thanks,
>>>
>>> Abhi
>>>
>>> On Thu, Feb 18, 2016 at 11:03 AM, moon soo Lee <mo...@apache.org> wrote:
>>>
>>>> Hi Abhi,
>>>>
>>>> Zeppelin 0.5.6-incubating binary package supposed to load installed
>>>> Spark binaries when SPARK_HOME is exported in conf/zeppelin-env.sh.
>>>>
>>>> Please let us know if it does not work with CDH 5.5.
>>>>
>>>> Thanks,
>>>> moon
>>>>
>>>>
>>>> On Wed, Feb 17, 2016 at 10:19 AM Abhi Basu <90...@gmail.com> wrote:
>>>>
>>>>> Will the binaries work right out of the box with CDH 5.5 and Spark
>>>>> 1.5.0 on YARN? Or, do I need to build from source as this describes:
>>>>> https://zeppelin.incubator.apache.org/docs/0.5.5-incubating/install/yarn_install.html
>>>>> ?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Abhi
>>>>>
>>>>> --
>>>>> Abhi Basu
>>>>>
>>>>
>>>
>>>
>>> --
>>> Abhi Basu
>>>
>>
>
>
> --
> Abhi Basu
>
Re: Zeppelin 0.56 binaries
Posted by Abhi Basu <90...@gmail.com>.
Moon:
Need a clarification. I am running off the binary package off Zeppelin 0.56
here on CDH 5.5.1. Spark and Hadoop paths are exported in zeppelin-env.sh.
Spark and Hive interpreters are configured. Ran a bunch of commands from
Zeppelin - Hive, PySpark and SparkSQL. Dont see a log for Spark interpreter
in the $ZEPPELIN_HOME/logs folder. I have also attached the yarn app log
and that shows Zeppelin is using Yarn. Where can I see the Spark logs?
I want to verify Zeppelin is working on Spark and YARN.
Thanks,
Abhi
On Thu, Feb 18, 2016 at 12:09 PM, moon soo Lee <mo...@apache.org> wrote:
> Right, after https://issues.apache.org/jira/browse/ZEPPELIN-160 resolved,
> Zeppelin supposed to work with any (supported) version of Spark without
> rebuild. Let us know if it does not work.
>
> Binary package is created by dev/create_release.sh script and you can
> check build flag
> https://github.com/apache/incubator-zeppelin/blob/v0.5.6/dev/create_release.sh#L128 used
> for the release.
>
> And thanks for pointing that install docs make confusion. I also think
> they need to be updated.
>
> Thanks,
> moon
>
> On Thu, Feb 18, 2016 at 12:01 PM Abhi Basu <90...@gmail.com> wrote:
>
>> Ok, thanks for the reply.
>>
>> So, from what you are saying, binary package should just use the
>> installed hadoop and spark versions, but just editing the zeppelin config
>> files - hadoop conf dir and spark home.
>>
>> Out of the box this would also support YARN, without being built with
>> -pYarn switch, as we had to before? Are the binaries build with that switc
>> already.
>>
>> Updating the install docs for binary would be a great way to lose some
>> confusion. I have been using Zeppelin for a while and earlier we had to
>> build from source specifying CDH, Spark versions and various switches.
>>
>> Just want to clarify before wasting time building from source :)
>>
>> BTW, awesome tool, love it :)
>>
>> Thanks,
>>
>> Abhi
>>
>> On Thu, Feb 18, 2016 at 11:03 AM, moon soo Lee <mo...@apache.org> wrote:
>>
>>> Hi Abhi,
>>>
>>> Zeppelin 0.5.6-incubating binary package supposed to load installed
>>> Spark binaries when SPARK_HOME is exported in conf/zeppelin-env.sh.
>>>
>>> Please let us know if it does not work with CDH 5.5.
>>>
>>> Thanks,
>>> moon
>>>
>>>
>>> On Wed, Feb 17, 2016 at 10:19 AM Abhi Basu <90...@gmail.com> wrote:
>>>
>>>> Will the binaries work right out of the box with CDH 5.5 and Spark
>>>> 1.5.0 on YARN? Or, do I need to build from source as this describes:
>>>> https://zeppelin.incubator.apache.org/docs/0.5.5-incubating/install/yarn_install.html
>>>> ?
>>>>
>>>> Thanks,
>>>>
>>>> Abhi
>>>>
>>>> --
>>>> Abhi Basu
>>>>
>>>
>>
>>
>> --
>> Abhi Basu
>>
>
--
Abhi Basu
Re: Zeppelin 0.56 binaries
Posted by moon soo Lee <mo...@apache.org>.
Right, after https://issues.apache.org/jira/browse/ZEPPELIN-160 resolved,
Zeppelin supposed to work with any (supported) version of Spark without
rebuild. Let us know if it does not work.
Binary package is created by dev/create_release.sh script and you can check
build flag
https://github.com/apache/incubator-zeppelin/blob/v0.5.6/dev/create_release.sh#L128
used
for the release.
And thanks for pointing that install docs make confusion. I also think they
need to be updated.
Thanks,
moon
On Thu, Feb 18, 2016 at 12:01 PM Abhi Basu <90...@gmail.com> wrote:
> Ok, thanks for the reply.
>
> So, from what you are saying, binary package should just use the installed
> hadoop and spark versions, but just editing the zeppelin config files -
> hadoop conf dir and spark home.
>
> Out of the box this would also support YARN, without being built with
> -pYarn switch, as we had to before? Are the binaries build with that switc
> already.
>
> Updating the install docs for binary would be a great way to lose some
> confusion. I have been using Zeppelin for a while and earlier we had to
> build from source specifying CDH, Spark versions and various switches.
>
> Just want to clarify before wasting time building from source :)
>
> BTW, awesome tool, love it :)
>
> Thanks,
>
> Abhi
>
> On Thu, Feb 18, 2016 at 11:03 AM, moon soo Lee <mo...@apache.org> wrote:
>
>> Hi Abhi,
>>
>> Zeppelin 0.5.6-incubating binary package supposed to load installed Spark
>> binaries when SPARK_HOME is exported in conf/zeppelin-env.sh.
>>
>> Please let us know if it does not work with CDH 5.5.
>>
>> Thanks,
>> moon
>>
>>
>> On Wed, Feb 17, 2016 at 10:19 AM Abhi Basu <90...@gmail.com> wrote:
>>
>>> Will the binaries work right out of the box with CDH 5.5 and Spark 1.5.0
>>> on YARN? Or, do I need to build from source as this describes:
>>> https://zeppelin.incubator.apache.org/docs/0.5.5-incubating/install/yarn_install.html
>>> ?
>>>
>>> Thanks,
>>>
>>> Abhi
>>>
>>> --
>>> Abhi Basu
>>>
>>
>
>
> --
> Abhi Basu
>
Re: Zeppelin 0.56 binaries
Posted by Abhi Basu <90...@gmail.com>.
Ok, thanks for the reply.
So, from what you are saying, binary package should just use the installed
hadoop and spark versions, but just editing the zeppelin config files -
hadoop conf dir and spark home.
Out of the box this would also support YARN, without being built with
-pYarn switch, as we had to before? Are the binaries build with that switc
already.
Updating the install docs for binary would be a great way to lose some
confusion. I have been using Zeppelin for a while and earlier we had to
build from source specifying CDH, Spark versions and various switches.
Just want to clarify before wasting time building from source :)
BTW, awesome tool, love it :)
Thanks,
Abhi
On Thu, Feb 18, 2016 at 11:03 AM, moon soo Lee <mo...@apache.org> wrote:
> Hi Abhi,
>
> Zeppelin 0.5.6-incubating binary package supposed to load installed Spark
> binaries when SPARK_HOME is exported in conf/zeppelin-env.sh.
>
> Please let us know if it does not work with CDH 5.5.
>
> Thanks,
> moon
>
>
> On Wed, Feb 17, 2016 at 10:19 AM Abhi Basu <90...@gmail.com> wrote:
>
>> Will the binaries work right out of the box with CDH 5.5 and Spark 1.5.0
>> on YARN? Or, do I need to build from source as this describes:
>> https://zeppelin.incubator.apache.org/docs/0.5.5-incubating/install/yarn_install.html
>> ?
>>
>> Thanks,
>>
>> Abhi
>>
>> --
>> Abhi Basu
>>
>
--
Abhi Basu
Re: Zeppelin 0.56 binaries
Posted by moon soo Lee <mo...@apache.org>.
Hi Abhi,
Zeppelin 0.5.6-incubating binary package supposed to load installed Spark
binaries when SPARK_HOME is exported in conf/zeppelin-env.sh.
Please let us know if it does not work with CDH 5.5.
Thanks,
moon
On Wed, Feb 17, 2016 at 10:19 AM Abhi Basu <90...@gmail.com> wrote:
> Will the binaries work right out of the box with CDH 5.5 and Spark 1.5.0
> on YARN? Or, do I need to build from source as this describes:
> https://zeppelin.incubator.apache.org/docs/0.5.5-incubating/install/yarn_install.html
> ?
>
> Thanks,
>
> Abhi
>
> --
> Abhi Basu
>