You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@griffin.apache.org by Darrell Taylor <da...@skybettingandgaming.com> on 2018/04/30 10:35:15 UTC
Disable Livy option?
Hi,
I was wondering if it’s feasible to disable the Livy option and have the ability to submit Spark jobs directly?
Having had a brief look through the code it seems like all the SparkConf and SparkContext is handled as normal, but couldn’t quite figure out where the job submission gets routed to Livy.
Thanks
--
Darrell Taylor
Principal Engineer (Data Science)
[cid:image001.png@01D3E077.4E8BEF40]
Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. The SKY trademarks are owned by Sky plc and are used under licence. Bonne Terre Limited (Alderney Co. No. 1110) with registered address Office 1, 1 The Crusher, Braye Harbour, Alderney GY9 3XX, Hestview Limited (England and Wales Co. No. 1100741) with registered address 2 Wellington Place, Leeds, LS1 4AP and Cyan Blue Odds Limited (Alderney Co. No. 1919) with registered address Century House, 12 Victoria Street, Alderney, GY9 3UF operate the Sky Betting and Gaming and Oddschecker services.
Re: Disable Livy option?
Posted by William Guo <gu...@apache.org>.
hi Taylor,
If I guess correctly, Usually you submit jobs by scp jars to cluster, ssh
to cluster and launch jobs by spart-submit CLI?
We will discuss whether/when to support this use case in our dev list.
Thanks,
William
On Mon, Apr 30, 2018 at 8:22 PM, Darrell Taylor <
darrell.taylor@skybettingandgaming.com> wrote:
> Hi William,
>
>
>
> Thanks for the info. I understand the potential firewall issues, we
> already addressed this for applications we deploy around the edge of our
> cluster.
>
>
>
> Currently the main reason for not using Livy is because it’s not currently
> available in our cluster.
>
>
>
> I was just curious how easy it would be to make it optional?
>
>
>
> Thanks
>
> Darrell.
>
>
>
>
>
> *From: *William Guo <gu...@apache.org>
> *Reply-To: *"users@griffin.apache.org" <us...@griffin.apache.org>
> *Date: *Monday, 30 April 2018 at 12:35
> *To: *"users@griffin.apache.org" <us...@griffin.apache.org>
> *Subject: *Re: Disable Livy option?
>
>
>
> Hi Taylor,
>
>
>
> The job submission is triggered by our job scheduler(quartz), by quartz,
> we periodically call livy service to submit spark jobs.
>
>
>
> We have thought submit jobs directly, but in many cases, that will cause
> firewall/security/auth issues.
>
>
>
> We leverage livy to submit jobs from http layer, because most cluster are
> accessible from http layer.
>
>
>
> What is your issue for using livy?
>
>
>
> Thanks,
>
> William
>
>
>
> On Mon, Apr 30, 2018 at 6:35 PM, Darrell Taylor <darrell.taylor@
> skybettingandgaming.com> wrote:
>
> Hi,
>
>
>
> I was wondering if it’s feasible to disable the Livy option and have the
> ability to submit Spark jobs directly?
>
>
>
> Having had a brief look through the code it seems like all the SparkConf
> and SparkContext is handled as normal, but couldn’t quite figure out where
> the job submission gets routed to Livy.
>
>
> Thanks
>
> --
>
>
> Information in this email including any attachments may be privileged,
> confidential and is intended exclusively for the addressee. The views
> expressed may not be official policy, but the personal views of the
> originator. If you have received it in error, please notify the sender by
> return e-mail and delete it from your system. You should not reproduce,
> distribute, store, retransmit, use or disclose its contents to anyone.
> Please note we reserve the right to monitor all e-mail communication
> through our internal and external networks. The SKY trademarks are owned by
> Sky plc and are used under licence. Bonne Terre Limited (Alderney Co. No.
> 1110) with registered address Office 1, 1 The Crusher, Braye Harbour,
> Alderney GY9 3XX, Hestview Limited (England and Wales Co. No. 1100741) with
> registered address 2 Wellington Place, Leeds, LS1 4AP
> <https://maps.google.com/?q=2+Wellington+Place,+Leeds,+LS1+4AP&entry=gmail&source=g>
> and Cyan Blue Odds Limited (Alderney Co. No. 1919) with registered address
> Century House, 12 Victoria Street, Alderney, GY9 3UF
> <https://maps.google.com/?q=12+Victoria+Street,+Alderney,+GY9+3UF&entry=gmail&source=g>
> operate the Sky Betting and Gaming and Oddschecker services.
>
Re: Disable Livy option?
Posted by Darrell Taylor <da...@skybettingandgaming.com>.
Hi William,
Thanks for the info. I understand the potential firewall issues, we already addressed this for applications we deploy around the edge of our cluster.
Currently the main reason for not using Livy is because it’s not currently available in our cluster.
I was just curious how easy it would be to make it optional?
Thanks
Darrell.
From: William Guo <gu...@apache.org>
Reply-To: "users@griffin.apache.org" <us...@griffin.apache.org>
Date: Monday, 30 April 2018 at 12:35
To: "users@griffin.apache.org" <us...@griffin.apache.org>
Subject: Re: Disable Livy option?
Hi Taylor,
The job submission is triggered by our job scheduler(quartz), by quartz, we periodically call livy service to submit spark jobs.
We have thought submit jobs directly, but in many cases, that will cause firewall/security/auth issues.
We leverage livy to submit jobs from http layer, because most cluster are accessible from http layer.
What is your issue for using livy?
Thanks,
William
On Mon, Apr 30, 2018 at 6:35 PM, Darrell Taylor <da...@skybettingandgaming.com>> wrote:
Hi,
I was wondering if it’s feasible to disable the Livy option and have the ability to submit Spark jobs directly?
Having had a brief look through the code it seems like all the SparkConf and SparkContext is handled as normal, but couldn’t quite figure out where the job submission gets routed to Livy.
Thanks
--
Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. The SKY trademarks are owned by Sky plc and are used under licence. Bonne Terre Limited (Alderney Co. No. 1110) with registered address Office 1, 1 The Crusher, Braye Harbour, Alderney GY9 3XX, Hestview Limited (England and Wales Co. No. 1100741) with registered address 2 Wellington Place, Leeds, LS1 4AP and Cyan Blue Odds Limited (Alderney Co. No. 1919) with registered address Century House, 12 Victoria Street, Alderney, GY9 3UF operate the Sky Betting and Gaming and Oddschecker services.
Re: Disable Livy option?
Posted by William Guo <gu...@apache.org>.
Hi Taylor,
The job submission is triggered by our job scheduler(quartz), by quartz, we
periodically call livy service to submit spark jobs.
We have thought submit jobs directly, but in many cases, that will cause
firewall/security/auth issues.
We leverage livy to submit jobs from http layer, because most cluster are
accessible from http layer.
What is your issue for using livy?
Thanks,
William
On Mon, Apr 30, 2018 at 6:35 PM, Darrell Taylor <
darrell.taylor@skybettingandgaming.com> wrote:
> Hi,
>
>
>
> I was wondering if it’s feasible to disable the Livy option and have the
> ability to submit Spark jobs directly?
>
>
>
> Having had a brief look through the code it seems like all the SparkConf
> and SparkContext is handled as normal, but couldn’t quite figure out where
> the job submission gets routed to Livy.
>
>
> Thanks
>
> --
>
> *Darrell Taylor*
>
> Principal Engineer (Data Science)
>
>
>
>
>
>
>
>
>
>
> Information in this email including any attachments may be privileged,
> confidential and is intended exclusively for the addressee. The views
> expressed may not be official policy, but the personal views of the
> originator. If you have received it in error, please notify the sender by
> return e-mail and delete it from your system. You should not reproduce,
> distribute, store, retransmit, use or disclose its contents to anyone.
> Please note we reserve the right to monitor all e-mail communication
> through our internal and external networks. The SKY trademarks are owned by
> Sky plc and are used under licence. Bonne Terre Limited (Alderney Co. No.
> 1110) with registered address Office 1, 1 The Crusher, Braye Harbour,
> Alderney GY9 3XX, Hestview Limited (England and Wales Co. No. 1100741) with
> registered address 2 Wellington Place, Leeds, LS1 4AP
> <https://maps.google.com/?q=2+Wellington+Place,+Leeds,+LS1+4AP&entry=gmail&source=g>
> and Cyan Blue Odds Limited (Alderney Co. No. 1919) with registered address
> Century House, 12 Victoria Street, Alderney, GY9 3UF
> <https://maps.google.com/?q=12+Victoria+Street,+Alderney,+GY9+3UF&entry=gmail&source=g>
> operate the Sky Betting and Gaming and Oddschecker services.
>