Apache Spark Operator for Kubernetes?

[Clayton Wohl <cl...@gmail.com>]

My company has been exploring the Google Spark Operator for running Spark
jobs on a Kubernetes cluster, but we've found lots of limitations and
problems, and the product seems weakly supported.

Is there any official Apache option, or plans for such an option, to run
Spark jobs on Kubernetes? Is there perhaps an official Apache Spark
Operator in the works?

We currently run jobs on both Databricks and on Amazon EMR, but it would be
nice to have a good option for running Spark directly on our Kubernetes
clusters.

thanks :)

Re: Apache Spark Operator for Kubernetes?

[Artemis User <ar...@dtechspace.com>]

If you have the hardware resources, it isn't difficult to set up Spark 
in a kubernetes cluster.  The online doc describes everything you would 
need (https://spark.apache.org/docs/latest/running-on-kubernetes.html).

You're right, both AWS EMR and Google's environment aren't flexible and 
not cheap.  At one time, we ended up spending over $1,800 per month on 
EMR.  If you have the hardware resources and know how configure and 
optimize your own OS and networks, going with an in-house solution will 
always be the best.

On 10/14/22 11:28 AM, Clayton Wohl wrote:
> My company has been exploring the Google Spark Operator for running 
> Spark jobs on a Kubernetes cluster, but we've found lots of 
> limitations and problems, and the product seems weakly supported.
>
> Is there any official Apache option, or plans for such an option, to 
> run Spark jobs on Kubernetes? Is there perhaps an official Apache 
> Spark Operator in the works?
>
> We currently run jobs on both Databricks and on Amazon EMR, but it 
> would be nice to have a good option for running Spark directly on our 
> Kubernetes clusters.
>
> thanks :)


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org