You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Paul Brenner <pb...@placeiq.com> on 2019/02/11 16:27:19 UTC
yarn cluster mode and dependencies
Hi All,
We are very excited about Zeppelin 0.81, especially the improved support for yarn cluster mode. One challenge we have run into so far is working with dependencies when using yarn cluster mode. For example if I set my interpreter with:
master: yarn
spark.submit.deployMode: cluster
zeppelin.spark.useNew: true
then in the dependencies section try to include something like:
com.maxmind.geoip2:geoip2:2.5.0
I find that when I try to import that package in zeppelin it gives me an error as if the dependency is not being included:
error: object maxmind is not a member of package com
Is there a different process required for loading dependencies in yarn cluster mode on zeppelin? Also has anyone seen any documentation or anything written from people working with yarn cluster mode on zeppelin that might have some guidance? It turns out that “zeppelin yarn cluster” are not the most unique terms for googling.
PAUL BRENNER
Sr. Data Scientist
pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
twitter @placeiq linkedin /placeiq
Re: yarn cluster mode and dependencies
Posted by Jeff Zhang <zj...@gmail.com>.
Hi Paul,
It seems a bug, someone else own found this issue.
https://jira.apache.org/jira/browse/ZEPPELIN-3986
I am looking at this issue.
Paul Brenner <pb...@placeiq.com> 于2019年2月12日周二 上午12:27写道：
> Hi All,
>
> We are very excited about Zeppelin 0.81, especially the improved support
> for yarn cluster mode. One challenge we have run into so far is working
> with dependencies when using yarn cluster mode. For example if I set my
> interpreter with:
>
> master: yarn
> spark.submit.deployMode: cluster
> zeppelin.spark.useNew: true
>
> then in the dependencies section try to include something like:
> com.maxmind.geoip2:geoip2:2.5.0
>
> I find that when I try to import that package in zeppelin it gives me an
> error as if the dependency is not being included:
>
> error: object maxmind is not a member of package com
>
> Is there a different process required for loading dependencies in yarn
> cluster mode on zeppelin? Also has anyone seen any documentation or
> anything written from people working with yarn cluster mode on zeppelin
> that might have some guidance? It turns out that “zeppelin yarn cluster”
> are not the most unique terms for googling.
>
>
> PAUL BRENNER
>
> Sr. Data Scientist
> pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq
> <https://www.linkedin.com/company/placeiq/>
> [image: PlaceIQ:Better Visits] <https://www.placeiq.com/>
>
--
Best Regards
Jeff Zhang