You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jagannath Majhi <ja...@cloud.cbnits.com> on 2024/02/19 13:36:59 UTC
Regarding Spark on Kubernetes(EKS)
Dear Spark Community,
I hope this email finds you well. I am reaching out to seek assistance and
guidance regarding a task I'm currently working on involving Apache Spark.
I have developed a JAR file that contains some Spark applications and
functionality, and I need to run this JAR file within a Spark cluster.
However, the JAR file is located in an AWS S3 bucket. I'm facing some
challenges in configuring Spark to access and execute this JAR file
directly from the S3 bucket.
I would greatly appreciate any advice, best practices, or pointers on how
to achieve this integration effectively. Specifically, I'm looking for
insights on:
1. Configuring Spark to access and retrieve the JAR file from an AWS S3
bucket.
2. Setting up the necessary permissions and authentication mechanisms to
ensure seamless access to the S3 bucket.
3. Any potential performance considerations or optimizations when
running Spark applications with dependencies stored in remote storage like
AWS S3.
If anyone in the community has prior experience or knowledge in this area,
I would be extremely grateful for your guidance. Additionally, if there are
any relevant resources, documentation, or tutorials that you could
recommend, it would be incredibly helpful.
Thank you very much for considering my request. I look forward to hearing
from you and benefiting from the collective expertise of the Spark
community.
Best regards, Jagannath Majhi
Re: Regarding Spark on Kubernetes(EKS)
Posted by Richard Smith <ri...@btinternet.com.INVALID>.
I run my Spark jobs in GCP with Google Dataproc using GCS buckets.
I've not used AWS, but its EMR product offers similar functionality to
Dataproc. The title of your post implies your Spark cluster runs on EKS.
You might be better off using EMR, see links below:
EMR
https://medium.com/big-data-on-amazon-elastic-mapreduce/run-a-spark-job-within-amazon-emr-in-15-minutes-68b02af1ae16
EKS https://medium.com/@vikas.navlani/running-spark-on-aws-eks-1cd4c31786c
Richard
On 19/02/2024 13:36, Jagannath Majhi wrote:
>
> Dear Spark Community,
>
> I hope this email finds you well. I am reaching out to seek assistance
> and guidance regarding a task I'm currently working on involving
> Apache Spark.
>
> I have developed a JAR file that contains some Spark applications and
> functionality, and I need to run this JAR file within a Spark cluster.
> However, the JAR file is located in an AWS S3 bucket. I'm facing some
> challenges in configuring Spark to access and execute this JAR file
> directly from the S3 bucket.
>
> I would greatly appreciate any advice, best practices, or pointers on
> how to achieve this integration effectively. Specifically, I'm looking
> for insights on:
>
> 1. Configuring Spark to access and retrieve the JAR file from an AWS
> S3 bucket.
> 2. Setting up the necessary permissions and authentication mechanisms
> to ensure seamless access to the S3 bucket.
> 3. Any potential performance considerations or optimizations when
> running Spark applications with dependencies stored in remote
> storage like AWS S3.
>
> If anyone in the community has prior experience or knowledge in this
> area, I would be extremely grateful for your guidance. Additionally,
> if there are any relevant resources, documentation, or tutorials that
> you could recommend, it would be incredibly helpful.
>
> Thank you very much for considering my request. I look forward to
> hearing from you and benefiting from the collective expertise of the
> Spark community.
>
> Best regards, Jagannath Majhi
>
Re: Regarding Spark on Kubernetes(EKS)
Posted by Jagannath Majhi <ja...@cloud.cbnits.com>.
Yes I have gone through it. So let's give me the setup. More context - My
jar file is in java language
On Mon, Feb 19, 2024, 8:53 PM Mich Talebzadeh <mi...@gmail.com>
wrote:
> Sure but first it would be beneficial to understand the way Spark works on
> Kubernetes and the concept.s
>
> Have a look at this article of mine
>
> Spark on Kubernetes, A Practitioner’s Guide
> <https://www.linkedin.com/pulse/spark-kubernetes-practitioners-guide-mich-talebzadeh-ph-d-%3FtrackingId=Wsu3lkoPaCWqGemYHe8%252BLQ%253D%253D/?trackingId=Wsu3lkoPaCWqGemYHe8%2BLQ%3D%3D>
>
> HTH
>
> Mich Talebzadeh,
> Dad | Technologist | Solutions Architect | Engineer
> London
> United Kingdom
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* The information provided is correct to the best of my
> knowledge but of course cannot be guaranteed . It is essential to note
> that, as with any advice, quote "one test result is worth one-thousand
> expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
> Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>
>
> On Mon, 19 Feb 2024 at 15:09, Jagannath Majhi <
> jagannath.majhi@cloud.cbnits.com> wrote:
>
>> Yes
>>
>> On Mon, Feb 19, 2024, 8:35 PM Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> OK you have a jar file that you want to work with when running using
>>> Spark on k8s as the execution engine (EKS) as opposed to YARN on EMR as
>>> the execution engine?
>>>
>>>
>>> Mich Talebzadeh,
>>> Dad | Technologist | Solutions Architect | Engineer
>>> London
>>> United Kingdom
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* The information provided is correct to the best of my
>>> knowledge but of course cannot be guaranteed . It is essential to note
>>> that, as with any advice, quote "one test result is worth one-thousand
>>> expert opinions (Werner
>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>>
>>>
>>> On Mon, 19 Feb 2024 at 14:38, Jagannath Majhi <
>>> jagannath.majhi@cloud.cbnits.com> wrote:
>>>
>>>> I am not using any private docker image. Only I am running the jar file
>>>> in EMR using spark-submit command so now I want to run this jar file in eks
>>>> so can you please tell me how can I set-up for this ??
>>>>
>>>> On Mon, Feb 19, 2024, 8:06 PM Jagannath Majhi <
>>>> jagannath.majhi@cloud.cbnits.com> wrote:
>>>>
>>>>> Can we connect over Google meet??
>>>>>
>>>>> On Mon, Feb 19, 2024, 8:03 PM Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> Where is your docker file? In ECR container registry.
>>>>>> If you are going to use EKS, then it need to be accessible to all
>>>>>> nodes of cluster
>>>>>>
>>>>>> When you build your docker image, put your jar under the $SPARK_HOME
>>>>>> directory. Then add a line to your docker build file as below
>>>>>> Here I am accessing Google BigQuery DW from EKS
>>>>>> # Add a BigQuery connector jar.
>>>>>> ENV SPARK_EXTRA_JARS_DIR=/opt/spark/jars/
>>>>>> ENV SPARK_EXTRA_CLASSPATH='/opt/spark/jars/*'
>>>>>> RUN mkdir -p "${SPARK_EXTRA_JARS_DIR}" \
>>>>>> && chown spark:spark "${SPARK_EXTRA_JARS_DIR}"
>>>>>> COPY --chown=spark:spark \
>>>>>> spark-bigquery-with-dependencies_2.12-0.22.2.jar
>>>>>> "${SPARK_EXTRA_JARS_DIR}"
>>>>>>
>>>>>> Here I am accessing Google BigQuery DW from EKS cluster
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>> Mich Talebzadeh,
>>>>>> Dad | Technologist | Solutions Architect | Engineer
>>>>>> London
>>>>>> United Kingdom
>>>>>>
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Disclaimer:* The information provided is correct to the best of my
>>>>>> knowledge but of course cannot be guaranteed . It is essential to note
>>>>>> that, as with any advice, quote "one test result is worth one-thousand
>>>>>> expert opinions (Werner
>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>>>>>
>>>>>>
>>>>>> On Mon, 19 Feb 2024 at 13:42, Jagannath Majhi <
>>>>>> jagannath.majhi@cloud.cbnits.com> wrote:
>>>>>>
>>>>>>> Dear Spark Community,
>>>>>>>
>>>>>>> I hope this email finds you well. I am reaching out to seek
>>>>>>> assistance and guidance regarding a task I'm currently working on involving
>>>>>>> Apache Spark.
>>>>>>>
>>>>>>> I have developed a JAR file that contains some Spark applications
>>>>>>> and functionality, and I need to run this JAR file within a Spark cluster.
>>>>>>> However, the JAR file is located in an AWS S3 bucket. I'm facing some
>>>>>>> challenges in configuring Spark to access and execute this JAR file
>>>>>>> directly from the S3 bucket.
>>>>>>>
>>>>>>> I would greatly appreciate any advice, best practices, or pointers
>>>>>>> on how to achieve this integration effectively. Specifically, I'm looking
>>>>>>> for insights on:
>>>>>>>
>>>>>>> 1. Configuring Spark to access and retrieve the JAR file from an
>>>>>>> AWS S3 bucket.
>>>>>>> 2. Setting up the necessary permissions and authentication
>>>>>>> mechanisms to ensure seamless access to the S3 bucket.
>>>>>>> 3. Any potential performance considerations or optimizations
>>>>>>> when running Spark applications with dependencies stored in remote storage
>>>>>>> like AWS S3.
>>>>>>>
>>>>>>> If anyone in the community has prior experience or knowledge in this
>>>>>>> area, I would be extremely grateful for your guidance. Additionally, if
>>>>>>> there are any relevant resources, documentation, or tutorials that you
>>>>>>> could recommend, it would be incredibly helpful.
>>>>>>>
>>>>>>> Thank you very much for considering my request. I look forward to
>>>>>>> hearing from you and benefiting from the collective expertise of the Spark
>>>>>>> community.
>>>>>>>
>>>>>>> Best regards, Jagannath Majhi
>>>>>>>
>>>>>>
Re: Regarding Spark on Kubernetes(EKS)
Posted by Mich Talebzadeh <mi...@gmail.com>.
Sure but first it would be beneficial to understand the way Spark works on
Kubernetes and the concept.s
Have a look at this article of mine
Spark on Kubernetes, A Practitioner’s Guide
<https://www.linkedin.com/pulse/spark-kubernetes-practitioners-guide-mich-talebzadeh-ph-d-%3FtrackingId=Wsu3lkoPaCWqGemYHe8%252BLQ%253D%253D/?trackingId=Wsu3lkoPaCWqGemYHe8%2BLQ%3D%3D>
HTH
Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
On Mon, 19 Feb 2024 at 15:09, Jagannath Majhi <
jagannath.majhi@cloud.cbnits.com> wrote:
> Yes
>
> On Mon, Feb 19, 2024, 8:35 PM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> OK you have a jar file that you want to work with when running using
>> Spark on k8s as the execution engine (EKS) as opposed to YARN on EMR as
>> the execution engine?
>>
>>
>> Mich Talebzadeh,
>> Dad | Technologist | Solutions Architect | Engineer
>> London
>> United Kingdom
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* The information provided is correct to the best of my
>> knowledge but of course cannot be guaranteed . It is essential to note
>> that, as with any advice, quote "one test result is worth one-thousand
>> expert opinions (Werner
>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>
>>
>> On Mon, 19 Feb 2024 at 14:38, Jagannath Majhi <
>> jagannath.majhi@cloud.cbnits.com> wrote:
>>
>>> I am not using any private docker image. Only I am running the jar file
>>> in EMR using spark-submit command so now I want to run this jar file in eks
>>> so can you please tell me how can I set-up for this ??
>>>
>>> On Mon, Feb 19, 2024, 8:06 PM Jagannath Majhi <
>>> jagannath.majhi@cloud.cbnits.com> wrote:
>>>
>>>> Can we connect over Google meet??
>>>>
>>>> On Mon, Feb 19, 2024, 8:03 PM Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> Where is your docker file? In ECR container registry.
>>>>> If you are going to use EKS, then it need to be accessible to all
>>>>> nodes of cluster
>>>>>
>>>>> When you build your docker image, put your jar under the $SPARK_HOME
>>>>> directory. Then add a line to your docker build file as below
>>>>> Here I am accessing Google BigQuery DW from EKS
>>>>> # Add a BigQuery connector jar.
>>>>> ENV SPARK_EXTRA_JARS_DIR=/opt/spark/jars/
>>>>> ENV SPARK_EXTRA_CLASSPATH='/opt/spark/jars/*'
>>>>> RUN mkdir -p "${SPARK_EXTRA_JARS_DIR}" \
>>>>> && chown spark:spark "${SPARK_EXTRA_JARS_DIR}"
>>>>> COPY --chown=spark:spark \
>>>>> spark-bigquery-with-dependencies_2.12-0.22.2.jar
>>>>> "${SPARK_EXTRA_JARS_DIR}"
>>>>>
>>>>> Here I am accessing Google BigQuery DW from EKS cluster
>>>>>
>>>>> HTH
>>>>>
>>>>> Mich Talebzadeh,
>>>>> Dad | Technologist | Solutions Architect | Engineer
>>>>> London
>>>>> United Kingdom
>>>>>
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> *Disclaimer:* The information provided is correct to the best of my
>>>>> knowledge but of course cannot be guaranteed . It is essential to note
>>>>> that, as with any advice, quote "one test result is worth one-thousand
>>>>> expert opinions (Werner
>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>>>>
>>>>>
>>>>> On Mon, 19 Feb 2024 at 13:42, Jagannath Majhi <
>>>>> jagannath.majhi@cloud.cbnits.com> wrote:
>>>>>
>>>>>> Dear Spark Community,
>>>>>>
>>>>>> I hope this email finds you well. I am reaching out to seek
>>>>>> assistance and guidance regarding a task I'm currently working on involving
>>>>>> Apache Spark.
>>>>>>
>>>>>> I have developed a JAR file that contains some Spark applications and
>>>>>> functionality, and I need to run this JAR file within a Spark cluster.
>>>>>> However, the JAR file is located in an AWS S3 bucket. I'm facing some
>>>>>> challenges in configuring Spark to access and execute this JAR file
>>>>>> directly from the S3 bucket.
>>>>>>
>>>>>> I would greatly appreciate any advice, best practices, or pointers on
>>>>>> how to achieve this integration effectively. Specifically, I'm looking for
>>>>>> insights on:
>>>>>>
>>>>>> 1. Configuring Spark to access and retrieve the JAR file from an
>>>>>> AWS S3 bucket.
>>>>>> 2. Setting up the necessary permissions and authentication
>>>>>> mechanisms to ensure seamless access to the S3 bucket.
>>>>>> 3. Any potential performance considerations or optimizations when
>>>>>> running Spark applications with dependencies stored in remote storage like
>>>>>> AWS S3.
>>>>>>
>>>>>> If anyone in the community has prior experience or knowledge in this
>>>>>> area, I would be extremely grateful for your guidance. Additionally, if
>>>>>> there are any relevant resources, documentation, or tutorials that you
>>>>>> could recommend, it would be incredibly helpful.
>>>>>>
>>>>>> Thank you very much for considering my request. I look forward to
>>>>>> hearing from you and benefiting from the collective expertise of the Spark
>>>>>> community.
>>>>>>
>>>>>> Best regards, Jagannath Majhi
>>>>>>
>>>>>
Re: Regarding Spark on Kubernetes(EKS)
Posted by Mich Talebzadeh <mi...@gmail.com>.
OK you have a jar file that you want to work with when running using Spark
on k8s as the execution engine (EKS) as opposed to YARN on EMR as the
execution engine?
Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
On Mon, 19 Feb 2024 at 14:38, Jagannath Majhi <
jagannath.majhi@cloud.cbnits.com> wrote:
> I am not using any private docker image. Only I am running the jar file in
> EMR using spark-submit command so now I want to run this jar file in eks so
> can you please tell me how can I set-up for this ??
>
> On Mon, Feb 19, 2024, 8:06 PM Jagannath Majhi <
> jagannath.majhi@cloud.cbnits.com> wrote:
>
>> Can we connect over Google meet??
>>
>> On Mon, Feb 19, 2024, 8:03 PM Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> Where is your docker file? In ECR container registry.
>>> If you are going to use EKS, then it need to be accessible to all nodes
>>> of cluster
>>>
>>> When you build your docker image, put your jar under the $SPARK_HOME
>>> directory. Then add a line to your docker build file as below
>>> Here I am accessing Google BigQuery DW from EKS
>>> # Add a BigQuery connector jar.
>>> ENV SPARK_EXTRA_JARS_DIR=/opt/spark/jars/
>>> ENV SPARK_EXTRA_CLASSPATH='/opt/spark/jars/*'
>>> RUN mkdir -p "${SPARK_EXTRA_JARS_DIR}" \
>>> && chown spark:spark "${SPARK_EXTRA_JARS_DIR}"
>>> COPY --chown=spark:spark \
>>> spark-bigquery-with-dependencies_2.12-0.22.2.jar
>>> "${SPARK_EXTRA_JARS_DIR}"
>>>
>>> Here I am accessing Google BigQuery DW from EKS cluster
>>>
>>> HTH
>>>
>>> Mich Talebzadeh,
>>> Dad | Technologist | Solutions Architect | Engineer
>>> London
>>> United Kingdom
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* The information provided is correct to the best of my
>>> knowledge but of course cannot be guaranteed . It is essential to note
>>> that, as with any advice, quote "one test result is worth one-thousand
>>> expert opinions (Werner
>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>>
>>>
>>> On Mon, 19 Feb 2024 at 13:42, Jagannath Majhi <
>>> jagannath.majhi@cloud.cbnits.com> wrote:
>>>
>>>> Dear Spark Community,
>>>>
>>>> I hope this email finds you well. I am reaching out to seek assistance
>>>> and guidance regarding a task I'm currently working on involving Apache
>>>> Spark.
>>>>
>>>> I have developed a JAR file that contains some Spark applications and
>>>> functionality, and I need to run this JAR file within a Spark cluster.
>>>> However, the JAR file is located in an AWS S3 bucket. I'm facing some
>>>> challenges in configuring Spark to access and execute this JAR file
>>>> directly from the S3 bucket.
>>>>
>>>> I would greatly appreciate any advice, best practices, or pointers on
>>>> how to achieve this integration effectively. Specifically, I'm looking for
>>>> insights on:
>>>>
>>>> 1. Configuring Spark to access and retrieve the JAR file from an
>>>> AWS S3 bucket.
>>>> 2. Setting up the necessary permissions and authentication
>>>> mechanisms to ensure seamless access to the S3 bucket.
>>>> 3. Any potential performance considerations or optimizations when
>>>> running Spark applications with dependencies stored in remote storage like
>>>> AWS S3.
>>>>
>>>> If anyone in the community has prior experience or knowledge in this
>>>> area, I would be extremely grateful for your guidance. Additionally, if
>>>> there are any relevant resources, documentation, or tutorials that you
>>>> could recommend, it would be incredibly helpful.
>>>>
>>>> Thank you very much for considering my request. I look forward to
>>>> hearing from you and benefiting from the collective expertise of the Spark
>>>> community.
>>>>
>>>> Best regards, Jagannath Majhi
>>>>
>>>
Re: Regarding Spark on Kubernetes(EKS)
Posted by Jagannath Majhi <ja...@cloud.cbnits.com>.
I am not using any private docker image. Only I am running the jar file in
EMR using spark-submit command so now I want to run this jar file in eks so
can you please tell me how can I set-up for this ??
On Mon, Feb 19, 2024, 8:06 PM Jagannath Majhi <
jagannath.majhi@cloud.cbnits.com> wrote:
> Can we connect over Google meet??
>
> On Mon, Feb 19, 2024, 8:03 PM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Where is your docker file? In ECR container registry.
>> If you are going to use EKS, then it need to be accessible to all nodes
>> of cluster
>>
>> When you build your docker image, put your jar under the $SPARK_HOME
>> directory. Then add a line to your docker build file as below
>> Here I am accessing Google BigQuery DW from EKS
>> # Add a BigQuery connector jar.
>> ENV SPARK_EXTRA_JARS_DIR=/opt/spark/jars/
>> ENV SPARK_EXTRA_CLASSPATH='/opt/spark/jars/*'
>> RUN mkdir -p "${SPARK_EXTRA_JARS_DIR}" \
>> && chown spark:spark "${SPARK_EXTRA_JARS_DIR}"
>> COPY --chown=spark:spark \
>> spark-bigquery-with-dependencies_2.12-0.22.2.jar
>> "${SPARK_EXTRA_JARS_DIR}"
>>
>> Here I am accessing Google BigQuery DW from EKS cluster
>>
>> HTH
>>
>> Mich Talebzadeh,
>> Dad | Technologist | Solutions Architect | Engineer
>> London
>> United Kingdom
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* The information provided is correct to the best of my
>> knowledge but of course cannot be guaranteed . It is essential to note
>> that, as with any advice, quote "one test result is worth one-thousand
>> expert opinions (Werner
>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>
>>
>> On Mon, 19 Feb 2024 at 13:42, Jagannath Majhi <
>> jagannath.majhi@cloud.cbnits.com> wrote:
>>
>>> Dear Spark Community,
>>>
>>> I hope this email finds you well. I am reaching out to seek assistance
>>> and guidance regarding a task I'm currently working on involving Apache
>>> Spark.
>>>
>>> I have developed a JAR file that contains some Spark applications and
>>> functionality, and I need to run this JAR file within a Spark cluster.
>>> However, the JAR file is located in an AWS S3 bucket. I'm facing some
>>> challenges in configuring Spark to access and execute this JAR file
>>> directly from the S3 bucket.
>>>
>>> I would greatly appreciate any advice, best practices, or pointers on
>>> how to achieve this integration effectively. Specifically, I'm looking for
>>> insights on:
>>>
>>> 1. Configuring Spark to access and retrieve the JAR file from an AWS
>>> S3 bucket.
>>> 2. Setting up the necessary permissions and authentication
>>> mechanisms to ensure seamless access to the S3 bucket.
>>> 3. Any potential performance considerations or optimizations when
>>> running Spark applications with dependencies stored in remote storage like
>>> AWS S3.
>>>
>>> If anyone in the community has prior experience or knowledge in this
>>> area, I would be extremely grateful for your guidance. Additionally, if
>>> there are any relevant resources, documentation, or tutorials that you
>>> could recommend, it would be incredibly helpful.
>>>
>>> Thank you very much for considering my request. I look forward to
>>> hearing from you and benefiting from the collective expertise of the Spark
>>> community.
>>>
>>> Best regards, Jagannath Majhi
>>>
>>
Re: Regarding Spark on Kubernetes(EKS)
Posted by Mich Talebzadeh <mi...@gmail.com>.
Where is your docker file? In ECR container registry.
If you are going to use EKS, then it need to be accessible to all nodes of
cluster
When you build your docker image, put your jar under the $SPARK_HOME
directory. Then add a line to your docker build file as below
Here I am accessing Google BigQuery DW from EKS
# Add a BigQuery connector jar.
ENV SPARK_EXTRA_JARS_DIR=/opt/spark/jars/
ENV SPARK_EXTRA_CLASSPATH='/opt/spark/jars/*'
RUN mkdir -p "${SPARK_EXTRA_JARS_DIR}" \
&& chown spark:spark "${SPARK_EXTRA_JARS_DIR}"
COPY --chown=spark:spark \
spark-bigquery-with-dependencies_2.12-0.22.2.jar
"${SPARK_EXTRA_JARS_DIR}"
Here I am accessing Google BigQuery DW from EKS cluster
HTH
Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
On Mon, 19 Feb 2024 at 13:42, Jagannath Majhi <
jagannath.majhi@cloud.cbnits.com> wrote:
> Dear Spark Community,
>
> I hope this email finds you well. I am reaching out to seek assistance and
> guidance regarding a task I'm currently working on involving Apache Spark.
>
> I have developed a JAR file that contains some Spark applications and
> functionality, and I need to run this JAR file within a Spark cluster.
> However, the JAR file is located in an AWS S3 bucket. I'm facing some
> challenges in configuring Spark to access and execute this JAR file
> directly from the S3 bucket.
>
> I would greatly appreciate any advice, best practices, or pointers on how
> to achieve this integration effectively. Specifically, I'm looking for
> insights on:
>
> 1. Configuring Spark to access and retrieve the JAR file from an AWS
> S3 bucket.
> 2. Setting up the necessary permissions and authentication mechanisms
> to ensure seamless access to the S3 bucket.
> 3. Any potential performance considerations or optimizations when
> running Spark applications with dependencies stored in remote storage like
> AWS S3.
>
> If anyone in the community has prior experience or knowledge in this area,
> I would be extremely grateful for your guidance. Additionally, if there are
> any relevant resources, documentation, or tutorials that you could
> recommend, it would be incredibly helpful.
>
> Thank you very much for considering my request. I look forward to hearing
> from you and benefiting from the collective expertise of the Spark
> community.
>
> Best regards, Jagannath Majhi
>