You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mark Elliot <ma...@theoremlp.com.INVALID> on 2023/08/09 15:37:42 UTC
dockerhub does not contain apache/spark-py 3.4.1
Hello,
I noticed that the apache/spark-py image for Spark's 3.4.1 release is not
available (apache/spark@3.4.1 is available). Would it be possible to get
the 3.4.1 release build for the apache/spark-py image published?
Thanks,
Mark
--
This communication, together with any attachments, is intended
only for the addressee(s) and may contain confidential, privileged or
proprietary information of Theorem Partners LLC ("Theorem"). By accepting
this communication you agree to keep confidential all information contained
in this communication, as well as any information derived by you from the
confidential information contained in this communication. Theorem does not
waive any confidentiality by misdelivery.
If you receive this
communication in error, any use, dissemination, printing or copying of all
or any part of it is strictly prohibited; please destroy all electronic and
paper copies and notify the sender immediately. Nothing in this email is
intended to constitute (1) investment, legal or tax advice, (2) any
recommendation to purchase or sell any security, (3) any advertisement or
offer of advisory services or (4) any offer to sell or solicitation of an
offer to buy any securities or other financial instrument in any
jurisdiction.
Theorem, including its agents or affiliates, reserves the
right to intercept, archive, monitor and review all communications to and
from its network, including this email and any email response to it.
Theorem makes no representation as to the accuracy or completeness of the
information in this communication and does not accept liability for any
errors or omissions in this communication, including any liability
resulting from its transmission by email, and undertakes no obligation to
update any information in this email or its attachments.
Re: dockerhub does not contain apache/spark-py 3.4.1
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Mark,
I created a spark3.4.1 docker file. Details from
spark-py-3.4.1-scala_2.12-11-jre-slim-buster
<https://hub.docker.com/repository/docker/michtalebzadeh/spark_dockerfiles/tags?page=1&ordering=last_updated>
Pull instructions are given
docker pull
michtalebzadeh/spark_dockerfiles:spark-py-3.4.1-scala_2.12-11-jre-slim-buster
It is 3.4.1 spark-py with no extra python packages
you can tag it as you wish
login to it as below
docker run -it
michtalebzadeh/spark_dockerfiles:spark-py-3.4.1-scala_2.12-11-jre-slim-buster
bash
185@b031a15c6730:/opt/spark/work-dir$ pip list
Package Version
------------- -------
asn1crypto 0.24.0
cryptography 2.6.1
entrypoints 0.3
keyring 17.1.1
keyrings.alt 3.1.1
pip 23.2.1
pycrypto 2.6.1
PyGObject 3.30.4
pyxdg 0.25
SecretStorage 2.3.1
setuptools 68.0.0
six 1.12.0
wheel 0.32.3
$SPARK_HOME/bin/spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.4.1
/_/
Using Scala version 2.12.17, OpenJDK 64-Bit Server VM, 11.0.11
Branch HEAD
Compiled by user centos on 2023-06-19T23:01:01Z
Revision 6b1ff22dde1ead51cbf370be6e48a802daae58b6
Url https://github.com/apache/spark
Built on java 11
185@b031a15c6730:/opt/spark/work-dir$ java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment 18.9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.11+9, mixed mode, sharing)
HTH
Mich Talebzadeh,
Solutions Architect/Engineering Lead
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Wed, 9 Aug 2023 at 17:41, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi Mark,
>
> you can build it yourself, no big deal :)
>
> REPOSITORY TAG
> IMAGE ID CREATED
> SIZE
> sparkpy/spark-py
> 3.4.1-scala_2.12-11-jre-slim-buster-Dockerfile a876102b2206 1
> second ago 1.09GB
> sparkpy/spark
> 3.4.1-scala_2.12-11-jre-slim-buster-Dockerfile 6f74f7475e01 3
> minutes ago 695MB
>
> Based on
>
> ARG java_image_tag=11-jre-slim ## java 11
> FROM openjdk:${java_image_tag}
>
> BASE_OS="buster"
> SPARK_VERSION="3.4.1"
> SCALA_VERSION="scala_2.12"
> DOCKERFILE="Dockerfile"
> DOCKERIMAGETAG="11-jre-slim"
>
> You need to modify the file
>
> $SPARK_HOME/kubernetes/dockerfiles/spark/Dockerfile
>
> and replace
>
> #ARG java_image_tag=17-jre
> #FROM eclipse-temurin:${java_image_tag}
>
> With
>
> ARG java_image_tag=11-jre-slim
> FROM openjdk:${java_image_tag}
>
> Which is Java 11
>
> HTH
>
>
> Mich Talebzadeh,
> Solutions Architect/Engineering Lead
> London
> United Kingdom
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Wed, 9 Aug 2023 at 16:43, Mark Elliot <ma...@theoremlp.com.invalid>
> wrote:
>
>> Hello,
>>
>> I noticed that the apache/spark-py image for Spark's 3.4.1 release is not
>> available (apache/spark@3.4.1 is available). Would it be possible to get
>> the 3.4.1 release build for the apache/spark-py image published?
>>
>> Thanks,
>>
>> Mark
>>
>> ------------------------------
>>
>> This communication, together with any attachments, is intended only for
>> the addressee(s) and may contain confidential, privileged or proprietary
>> information of Theorem Partners LLC ("Theorem"). By accepting this
>> communication you agree to keep confidential all information contained in
>> this communication, as well as any information derived by you from the
>> confidential information contained in this communication. Theorem does not
>> waive any confidentiality by misdelivery.
>>
>> If you receive this communication in error, any use, dissemination,
>> printing or copying of all or any part of it is strictly prohibited; please
>> destroy all electronic and paper copies and notify the sender immediately.
>> Nothing in this email is intended to constitute (1) investment, legal or
>> tax advice, (2) any recommendation to purchase or sell any security, (3)
>> any advertisement or offer of advisory services or (4) any offer to sell or
>> solicitation of an offer to buy any securities or other financial
>> instrument in any jurisdiction.
>>
>> Theorem, including its agents or affiliates, reserves the right to
>> intercept, archive, monitor and review all communications to and from its
>> network, including this email and any email response to it.
>>
>> Theorem makes no representation as to the accuracy or completeness of the
>> information in this communication and does not accept liability for any
>> errors or omissions in this communication, including any liability
>> resulting from its transmission by email, and undertakes no obligation to
>> update any information in this email or its attachments.
>>
>
Re: dockerhub does not contain apache/spark-py 3.4.1
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Mark,
you can build it yourself, no big deal :)
REPOSITORY TAG
IMAGE ID CREATED
SIZE
sparkpy/spark-py
3.4.1-scala_2.12-11-jre-slim-buster-Dockerfile a876102b2206 1
second ago 1.09GB
sparkpy/spark
3.4.1-scala_2.12-11-jre-slim-buster-Dockerfile 6f74f7475e01 3
minutes ago 695MB
Based on
ARG java_image_tag=11-jre-slim ## java 11
FROM openjdk:${java_image_tag}
BASE_OS="buster"
SPARK_VERSION="3.4.1"
SCALA_VERSION="scala_2.12"
DOCKERFILE="Dockerfile"
DOCKERIMAGETAG="11-jre-slim"
You need to modify the file
$SPARK_HOME/kubernetes/dockerfiles/spark/Dockerfile
and replace
#ARG java_image_tag=17-jre
#FROM eclipse-temurin:${java_image_tag}
With
ARG java_image_tag=11-jre-slim
FROM openjdk:${java_image_tag}
Which is Java 11
HTH
Mich Talebzadeh,
Solutions Architect/Engineering Lead
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Wed, 9 Aug 2023 at 16:43, Mark Elliot <ma...@theoremlp.com.invalid> wrote:
> Hello,
>
> I noticed that the apache/spark-py image for Spark's 3.4.1 release is not
> available (apache/spark@3.4.1 is available). Would it be possible to get
> the 3.4.1 release build for the apache/spark-py image published?
>
> Thanks,
>
> Mark
>
> ------------------------------
>
> This communication, together with any attachments, is intended only for
> the addressee(s) and may contain confidential, privileged or proprietary
> information of Theorem Partners LLC ("Theorem"). By accepting this
> communication you agree to keep confidential all information contained in
> this communication, as well as any information derived by you from the
> confidential information contained in this communication. Theorem does not
> waive any confidentiality by misdelivery.
>
> If you receive this communication in error, any use, dissemination,
> printing or copying of all or any part of it is strictly prohibited; please
> destroy all electronic and paper copies and notify the sender immediately.
> Nothing in this email is intended to constitute (1) investment, legal or
> tax advice, (2) any recommendation to purchase or sell any security, (3)
> any advertisement or offer of advisory services or (4) any offer to sell or
> solicitation of an offer to buy any securities or other financial
> instrument in any jurisdiction.
>
> Theorem, including its agents or affiliates, reserves the right to
> intercept, archive, monitor and review all communications to and from its
> network, including this email and any email response to it.
>
> Theorem makes no representation as to the accuracy or completeness of the
> information in this communication and does not accept liability for any
> errors or omissions in this communication, including any liability
> resulting from its transmission by email, and undertakes no obligation to
> update any information in this email or its attachments.
>