You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by purna pradeep <pu...@gmail.com> on 2018/03/21 02:47:24 UTC
Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
,now i want to run spark-submit from AWS lambda function to k8s
master,would like to know if there is any REST interface to run Spark
submit on k8s Master
Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Posted by Josh Goldsborough <jo...@gmail.com>.
Purna,
It's a bit tangental to your original question but heads up that Amazon EKS
is in Preview right now:
https://aws.amazon.com/eks/
I don't know if it actually allows a nice interface between k8s hosted
Spark & Lamda functions (my suspicion is it won't fix your problem), but
might be something worth taking a peak at, since you're already doing AWS
hosted K8s stuff.
On Wed, Mar 21, 2018 at 9:37 AM, lucas.gary@gmail.com <lu...@gmail.com>
wrote:
> Speaking from experience, if you're already operating a kubernetes
> cluster. Getting a spark workload operating there is nearly an order of
> magnitude simpler than working with / around EMR.
>
> That's not say EMR is excessively hard, just that Kubernetes is easier,
> all the steps to getting your application deployed are well documented and
> ultimately the whole process is more visible.
>
> Also, thanks for the link Yinan! I'll be investigating that project!
>
> We have in the past used EMR for larger workloads and as soon as we
> announced our users could run those workloads on our k8s cluster everyone
> immediately moved their workloads over. This was despite having Spark on
> K8s still in an infant state. No one has expressed interest in moving back.
>
> G
>
> On 21 March 2018 at 07:32, Gourav Sengupta <go...@gmail.com>
> wrote:
>
>> Hi,
>>
>> just out of curiosity, but since it in AWS, is there any specific reason
>> not to use EMR? Or any particular reason to use Kubernetes?
>>
>>
>> Regards,
>> Gourav Sengupta
>>
>> On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep <pu...@gmail.com>
>> wrote:
>>
>>> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
>>> ,now i want to run spark-submit from AWS lambda function to k8s
>>> master,would like to know if there is any REST interface to run Spark
>>> submit on k8s Master
>>
>>
>>
>
Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Posted by Gourav Sengupta <go...@gmail.com>.
Hi Lucas,
Thanks a ton for responding.
have you used livy and SPARK in EMR? I am genuinely not sure how adding a
spark-submit in EMR is hard, it is just one line of code.
I must be missing something here
Regards,
Gourav Sengupta
On Wed, Mar 21, 2018 at 2:37 PM, lucas.gary@gmail.com <lu...@gmail.com>
wrote:
> Speaking from experience, if you're already operating a kubernetes
> cluster. Getting a spark workload operating there is nearly an order of
> magnitude simpler than working with / around EMR.
>
> That's not say EMR is excessively hard, just that Kubernetes is easier,
> all the steps to getting your application deployed are well documented and
> ultimately the whole process is more visible.
>
> Also, thanks for the link Yinan! I'll be investigating that project!
>
> We have in the past used EMR for larger workloads and as soon as we
> announced our users could run those workloads on our k8s cluster everyone
> immediately moved their workloads over. This was despite having Spark on
> K8s still in an infant state. No one has expressed interest in moving back.
>
> G
>
> On 21 March 2018 at 07:32, Gourav Sengupta <go...@gmail.com>
> wrote:
>
>> Hi,
>>
>> just out of curiosity, but since it in AWS, is there any specific reason
>> not to use EMR? Or any particular reason to use Kubernetes?
>>
>>
>> Regards,
>> Gourav Sengupta
>>
>> On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep <pu...@gmail.com>
>> wrote:
>>
>>> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
>>> ,now i want to run spark-submit from AWS lambda function to k8s
>>> master,would like to know if there is any REST interface to run Spark
>>> submit on k8s Master
>>
>>
>>
>
Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Posted by "lucas.gary@gmail.com" <lu...@gmail.com>.
Speaking from experience, if you're already operating a kubernetes
cluster. Getting a spark workload operating there is nearly an order of
magnitude simpler than working with / around EMR.
That's not say EMR is excessively hard, just that Kubernetes is easier, all
the steps to getting your application deployed are well documented and
ultimately the whole process is more visible.
Also, thanks for the link Yinan! I'll be investigating that project!
We have in the past used EMR for larger workloads and as soon as we
announced our users could run those workloads on our k8s cluster everyone
immediately moved their workloads over. This was despite having Spark on
K8s still in an infant state. No one has expressed interest in moving back.
G
On 21 March 2018 at 07:32, Gourav Sengupta <go...@gmail.com>
wrote:
> Hi,
>
> just out of curiosity, but since it in AWS, is there any specific reason
> not to use EMR? Or any particular reason to use Kubernetes?
>
>
> Regards,
> Gourav Sengupta
>
> On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep <pu...@gmail.com>
> wrote:
>
>> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
>> ,now i want to run spark-submit from AWS lambda function to k8s
>> master,would like to know if there is any REST interface to run Spark
>> submit on k8s Master
>
>
>
Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Posted by Gourav Sengupta <go...@gmail.com>.
Hi,
just out of curiosity, but since it in AWS, is there any specific reason
not to use EMR? Or any particular reason to use Kubernetes?
Regards,
Gourav Sengupta
On Wed, Mar 21, 2018 at 2:47 AM, purna pradeep <pu...@gmail.com>
wrote:
> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
> ,now i want to run spark-submit from AWS lambda function to k8s
> master,would like to know if there is any REST interface to run Spark
> submit on k8s Master
Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Posted by purna pradeep <pu...@gmail.com>.
Thanks Yinan,
Looks like this is stil in alpha version.
Would like to know if there is any rest-interface for spark2.3 job
submission similar to spark 2.2 as I need to submit spark applications to
k8 master based on different events (cron or s3 file based trigger)
On Tue, Mar 20, 2018 at 11:50 PM Yinan Li <li...@gmail.com> wrote:
> One option is the Spark Operator
> <https://github.com/GoogleCloudPlatform/spark-on-k8s-operator>. It allows
> specifying and running Spark applications on Kubernetes using Kubernetes
> custom resources objects. It takes SparkApplication CRD objects and
> automatically submits the applications to run on a Kubernetes cluster.
>
> Yinan
>
> On Tue, Mar 20, 2018 at 7:47 PM, purna pradeep <pu...@gmail.com>
> wrote:
>
>> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
>> ,now i want to run spark-submit from AWS lambda function to k8s
>> master,would like to know if there is any REST interface to run Spark
>> submit on k8s Master
>
>
>
Re: Rest API for Spark2.3 submit on kubernetes(version 1.8.*) cluster
Posted by Yinan Li <li...@gmail.com>.
One option is the Spark Operator
<https://github.com/GoogleCloudPlatform/spark-on-k8s-operator>. It allows
specifying and running Spark applications on Kubernetes using Kubernetes
custom resources objects. It takes SparkApplication CRD objects and
automatically submits the applications to run on a Kubernetes cluster.
Yinan
On Tue, Mar 20, 2018 at 7:47 PM, purna pradeep <pu...@gmail.com>
wrote:
> Im using kubernetes cluster on AWS to run spark jobs ,im using spark 2.3
> ,now i want to run spark-submit from AWS lambda function to k8s
> master,would like to know if there is any REST interface to run Spark
> submit on k8s Master