You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mohammad Abdollahzade Arani <ma...@gmail.com> on 2022/10/15 07:26:47 UTC
spark on kubernetes
I have a k8s cluster and a spark cluster.
my question is is as bellow:
https://stackoverflow.com/questions/74053948/how-to-resolve-pods-is-forbidden-user-systemanonymous-cannot-watch-resourc
I have searched and I found lot's of other similar questions on
stackoverflow without an answer like bellow:
https://stackoverflow.com/questions/61982896/how-to-fix-pods-is-forbidden-user-systemanonymous-cannot-watch-resource
--
Best Wishes!
Mohammad Abdollahzade Arani
Computer Engineering @ SBU
Re: spark on kubernetes
Posted by Qian Sun <qi...@gmail.com>.
Glad to hear it!
On Sun, Oct 16, 2022 at 2:37 PM Mohammad Abdollahzade Arani <
mamadazarani@gmail.com> wrote:
> Hi Qian,
> Thanks for the reply and I'm So sorry for the late reply.
> I found the answer. My mistake was token conversion. I had to decode
> base64 the service accounts token and certificate.
> and you are right I have to use `service account cert` to configure
> spark.kubernetes.authenticate.caCertFile.
> Thanks again. best regards.
>
> On Sat, Oct 15, 2022 at 4:51 PM Qian Sun <qi...@gmail.com> wrote:
>
>> Hi Mohammad
>> Did you try this command?
>>
>>
>> ./bin/spark-submit \ --master k8s://https://vm13:6443 \ --class com.example.WordCounter \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=default \ --conf spark.kubernetes.container.image=private-docker-registery/spark/spark:3.2.1-3 \ --conf spark.kubernetes.namespace=default \ java-word-count-1.0-SNAPSHOT.jar
>>
>> If you want spark.kubernetes.authenticate.caCertFile, you need to
>> configure it to serviceaccount certFile instead of apiserver certFile.
>>
>> On Sat, Oct 15, 2022 at 8:30 PM Mohammad Abdollahzade Arani
>> mamadazarani@gmail.com <ht...@gmail.com> wrote:
>>
>> I have a k8s cluster and a spark cluster.
>>> my question is is as bellow:
>>>
>>>
>>> https://stackoverflow.com/questions/74053948/how-to-resolve-pods-is-forbidden-user-systemanonymous-cannot-watch-resourc
>>>
>>> I have searched and I found lot's of other similar questions on
>>> stackoverflow without an answer like bellow:
>>>
>>>
>>> https://stackoverflow.com/questions/61982896/how-to-fix-pods-is-forbidden-user-systemanonymous-cannot-watch-resource
>>>
>>>
>>> --
>>> Best Wishes!
>>> Mohammad Abdollahzade Arani
>>> Computer Engineering @ SBU
>>>
>>> --
>> Best!
>> Qian Sun
>>
>
>
> --
> Best Wishes!
> Mohammad Abdollahzade Arani
> Computer Engineering @ SBU
>
>
--
Best!
Qian Sun
Re: spark on kubernetes
Posted by Qian Sun <qi...@gmail.com>.
Hi Mohammad
Did you try this command?
./bin/spark-submit \ --master k8s://https://vm13:6443 \
--class com.example.WordCounter \ --conf
spark.kubernetes.authenticate.driver.serviceAccountName=default \
--conf spark.kubernetes.container.image=private-docker-registery/spark/spark:3.2.1-3
\ --conf spark.kubernetes.namespace=default \
java-word-count-1.0-SNAPSHOT.jar
If you want spark.kubernetes.authenticate.caCertFile, you need to configure
it to serviceaccount certFile instead of apiserver certFile.
On Sat, Oct 15, 2022 at 8:30 PM Mohammad Abdollahzade Arani
mamadazarani@gmail.com <ht...@gmail.com> wrote:
I have a k8s cluster and a spark cluster.
> my question is is as bellow:
>
>
> https://stackoverflow.com/questions/74053948/how-to-resolve-pods-is-forbidden-user-systemanonymous-cannot-watch-resourc
>
> I have searched and I found lot's of other similar questions on
> stackoverflow without an answer like bellow:
>
>
> https://stackoverflow.com/questions/61982896/how-to-fix-pods-is-forbidden-user-systemanonymous-cannot-watch-resource
>
>
> --
> Best Wishes!
> Mohammad Abdollahzade Arani
> Computer Engineering @ SBU
>
> --
Best!
Qian Sun