You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Paul Brenner <pb...@placeiq.com> on 2017/12/28 18:22:01 UTC
Setting spark.driver.memory
We are on Zeppelin 0.7.2 and it I suspect that we are unable to set spark.driver.memory through the interpreter settings page.
If I add a property “spark.driver.memory” to my interpreter and set it to 10g, I see in my spark application environment page that the spark property spark.driver.memory is intact set to 10g, but when I look at the executors tab I see my driver is only getting 384mb (which seems suspiciously like the amount I would get if I was actually getting a default 512mb).
Digging around the web it sounds like people have had luck setting driver memory conf/zeppelin-env.sh ( e.g., http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Can-not-configure-driver-memory-size-td1513.html ( https://share.polymail.io/v1/z/b/NWE0NTMzZTJiNDk5/fb8ILoEIDvV495bIeoxmli2x06dNLmX9fdn32TWm4r_2E_eb58iBHrOUgR1IdZZV5FA-zX-LnlntRr0sDd2hp3rCP7C4TNKHHi8iK237Xxa_bttU1hPUHJs_j2Klm98jUhdvxiEVnEc2fSr903Dopc59FnamRjVVKhq4Macg3YCEdTaCMwd4XD9J_otoNQsgWqltSPWYHRfEeGZQjWysjJH1msfX50DAwC7CzDqSHyW5BqqN3uPCIAP8KiG_aoY9THg5KTObDvjZAPs1hy_pX9W3cNDtXaVw2-Bb-5X3wyFf6xIJIbunaWVNSP3qxw== ) ).
* Can anyone confirm whether it is possible to set driver memory in interpreter settings?
* If it isn’t possible, how can I find out which settings are settable through interpreter settings and which aren’t?
* Also, if it isn’t possible, should I open a ticket for this? We really want users to be able to customize driver memory size.
( http://www.placeiq.com/ ) ( http://www.placeiq.com/ ) ( http://www.placeiq.com/ ) *Paul Brenner* ( https://twitter.com/placeiq ) ( https://twitter.com/placeiq ) ( https://twitter.com/placeiq ) ( https://www.facebook.com/PlaceIQ ) ( https://www.facebook.com/PlaceIQ ) ( https://www.linkedin.com/company/placeiq ) ( https://www.linkedin.com/company/placeiq ) DATA SCIENTIST (217) 390-3033
( http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/ ) ( http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ ) ( http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ ) ( http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ ) ( http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ ) ( http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/ ) ( http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ ) ( http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ ) ( http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ ) ( http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ ) ( http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ ) ( http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP ) ( http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/ ) ( http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/ ) ( http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/ ) ( http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/ ) ( https://www.placeiq.com/2017/05/placeiqs-landmark-powers-location-based-insight-innovation-for-ansible-gstv-havas-media-the-media-kitchen-and-more/ ) PlaceIQ:CES 2018 ( http://pages.placeiq.com/CES2018_MeetingRequest.html )
Re: Setting spark.driver.memory
Posted by Paul Brenner <pb...@placeiq.com>.
Thanks for the quick response!
Seeing as the other live thread in this group is about how we can’t build 0.8.0 anymore due to the dependency convergence issues, do I have any options in the near term? Do I just need to wait until 0.8 comes out in some unspecified number of months? Have people had success resolving these issues and getting 0.8 to build? I hit a dead end last time I tried and have been struggling to get time to dedicate to trying again.
( http://www.placeiq.com/ ) ( http://www.placeiq.com/ ) ( http://www.placeiq.com/ ) *Paul Brenner* ( https://twitter.com/placeiq ) ( https://twitter.com/placeiq ) ( https://twitter.com/placeiq ) ( https://www.facebook.com/PlaceIQ ) ( https://www.facebook.com/PlaceIQ ) ( https://www.linkedin.com/company/placeiq ) ( https://www.linkedin.com/company/placeiq ) DATA SCIENTIST (217) 390-3033
( http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/ ) ( http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ ) ( http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ ) ( http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ ) ( http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ ) ( http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/ ) ( http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ ) ( http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ ) ( http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ ) ( http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ ) ( http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ ) ( http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP ) ( http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/ ) ( http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/ ) ( http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/ ) ( http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/ ) ( https://www.placeiq.com/2017/05/placeiqs-landmark-powers-location-based-insight-innovation-for-ansible-gstv-havas-media-the-media-kitchen-and-more/ ) PlaceIQ:CES 2018 ( http://pages.placeiq.com/CES2018_MeetingRequest.html )
On Thu, Dec 28, 2017 at 5:58 PM Jeff Zhang < Jeff Zhang ( Jeff Zhang <zj...@gmail.com> ) > wrote:
>
>
> This is a known issue, and fixed in 0.8.0
>
>
> https://issues.apache.org/jira/browse/ZEPPELIN-1263
>
>
>
>
>
>
> Paul Brenner < pbrenner@placeiq.com >于2017年12月29日周五 上午2:22写道：
>
>
>>
>> We are on Zeppelin 0.7.2 and it I suspect that we are unable to set
>> spark.driver.memory through the interpreter settings page.
>>
>>
>> If I add a property “spark.driver.memory” to my interpreter and set it to
>> 10g, I see in my spark application environment page that the spark
>> property spark.driver.memory is intact set to 10g, but when I look at the
>> executors tab I see my driver is only getting 384mb (which seems
>> suspiciously like the amount I would get if I was actually getting a
>> default 512mb).
>>
>>
>> Digging around the web it sounds like people have had luck setting driver
>> memory conf/zeppelin-env.sh ( e.g., http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Can-not-configure-driver-memory-size-td1513.html
>> (
>> https://share.polymail.io/v1/z/b/NWE0NTMzZTJiNDk5/fb8ILoEIDvV495bIeoxmli2x06dNLmX9fdn32TWm4r_2E_eb58iBHrOUgR1IdZZV5FA-zX-LnlntRr0sDd2hp3rCP7C4TNKHHi8iK237Xxa_bttU1hPUHJs_j2Klm98jUhdvxiEVnEc2fSr903Dopc59FnamRjVVKhq4Macg3YCEdTaCMwd4XD9J_otoNQsgWqltSPWYHRfEeGZQjWysjJH1msfX50DAwC7CzDqSHyW5BqqN3uPCIAP8KiG_aoY9THg5KTObDvjZAPs1hy_pX9W3cNDtXaVw2-Bb-5X3wyFf6xIJIbunaWVNSP3qxw==
>> ) ).
>>
>>
>>
>>
>> * Can anyone confirm whether it is possible to set driver memory in
>> interpreter settings?
>> * If it isn’t possible, how can I find out which settings are settable
>> through interpreter settings and which aren’t?
>> * Also, if it isn’t possible, should I open a ticket for this? We really
>> want users to be able to customize driver memory size.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ( http://www.placeiq.com/ ) ( http://www.placeiq.com/ ) (
>> http://www.placeiq.com/ ) *Paul Brenner* ( https://twitter.com/placeiq ) (
>> https://twitter.com/placeiq ) ( https://twitter.com/placeiq ) (
>> https://www.facebook.com/PlaceIQ ) ( https://www.facebook.com/PlaceIQ ) (
>> https://www.linkedin.com/company/placeiq ) (
>> https://www.linkedin.com/company/placeiq ) DATA SCIENTIST (217) 390-3033 (
>> tel:(217)%20390-3033 )
>>
>>
>>
>>
>> (
>> http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/
>> ) (
>> http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/
>> ) (
>> http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/
>> ) (
>> http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/
>> ) (
>> http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/
>> ) (
>> http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/
>> ) (
>> http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/
>> ) (
>> http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/
>> ) (
>> http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/
>> ) (
>> http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/
>> ) (
>> http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/
>> ) (
>> http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP
>> ) (
>> http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/
>> ) (
>> http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/
>> ) (
>> http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/
>> ) (
>> http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/
>> ) (
>> https://www.placeiq.com/2017/05/placeiqs-landmark-powers-location-based-insight-innovation-for-ansible-gstv-havas-media-the-media-kitchen-and-more/
>> ) PlaceIQ:CES 2018 ( http://pages.placeiq.com/CES2018_MeetingRequest.html )
>>
>>
>>
>>
>
>
Re: Setting spark.driver.memory
Posted by Jeff Zhang <zj...@gmail.com>.
This is a known issue, and fixed in 0.8.0
https://issues.apache.org/jira/browse/ZEPPELIN-1263
Paul Brenner <pb...@placeiq.com>于2017年12月29日周五 上午2:22写道：
> We are on Zeppelin 0.7.2 and it I suspect that we are unable to set
> spark.driver.memory through the interpreter settings page.
>
> If I add a property “spark.driver.memory” to my interpreter and set it to
> 10g, I see in my spark application environment page that the spark property
> spark.driver.memory is intact set to 10g, but when I look at the executors
> tab I see my driver is only getting 384mb (which seems suspiciously like
> the amount I would get if I was actually getting a default 512mb).
>
> Digging around the web it sounds like people have had luck setting driver
> memory conf/zeppelin-env.sh ( e.g.,
> http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Can-not-configure-driver-memory-size-td1513.html
> <https://share.polymail.io/v1/z/b/NWE0NTMzZTJiNDk5/fb8ILoEIDvV495bIeoxmli2x06dNLmX9fdn32TWm4r_2E_eb58iBHrOUgR1IdZZV5FA-zX-LnlntRr0sDd2hp3rCP7C4TNKHHi8iK237Xxa_bttU1hPUHJs_j2Klm98jUhdvxiEVnEc2fSr903Dopc59FnamRjVVKhq4Macg3YCEdTaCMwd4XD9J_otoNQsgWqltSPWYHRfEeGZQjWysjJH1msfX50DAwC7CzDqSHyW5BqqN3uPCIAP8KiG_aoY9THg5KTObDvjZAPs1hy_pX9W3cNDtXaVw2-Bb-5X3wyFf6xIJIbunaWVNSP3qxw==>
> ).
>
>
> - Can anyone confirm whether it is possible to set driver memory in
> interpreter settings?
> - If it isn’t possible, how can I find out which settings are settable
> through interpreter settings and which aren’t?
> - Also, if it isn’t possible, should I open a ticket for this? We
> really want users to be able to customize driver memory size.
>
>
>
>
>
> <http://www.placeiq.com/> <http://www.placeiq.com/>
> <http://www.placeiq.com/> *Paul Brenner* <https://twitter.com/placeiq>
> <https://twitter.com/placeiq> <https://twitter.com/placeiq>
> <https://www.facebook.com/PlaceIQ> <https://www.facebook.com/PlaceIQ>
> <https://www.linkedin.com/company/placeiq>
> <https://www.linkedin.com/company/placeiq>
> DATA SCIENTIST
> *(217) 390-3033 <(217)%20390-3033> *
>
> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP>
> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
> <https://www.placeiq.com/2017/05/placeiqs-landmark-powers-location-based-insight-innovation-for-ansible-gstv-havas-media-the-media-kitchen-and-more/>[image:
> PlaceIQ:CES 2018] <http://pages.placeiq.com/CES2018_MeetingRequest.html>
>