You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by GitBox <gi...@apache.org> on 2022/06/18 18:37:54 UTC

[GitHub] [spark-website] holdenk opened a new pull request, #400: [SPARK-39512] Document docker image release steps

holdenk opened a new pull request, #400:
URL: https://github.com/apache/spark-website/pull/400

   Document the docker image release steps for the release manager to follow when finalizing the release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] holdenk commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
holdenk commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1174357233

   My bad let me pick this up tomorrow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] srowen commented on a diff in pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
srowen commented on code in PR #400:
URL: https://github.com/apache/spark-website/pull/400#discussion_r901012063


##########
site/sitemap.xml:
##########
@@ -941,27 +941,27 @@
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/graphx/</loc>
+  <loc>https://spark.apache.org/news/</loc>

Review Comment:
   I don't know which ordering is correct, but maybe revert this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] gengliangwang commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1166188886

   BTW, I think we should add the docker images into the RC vote email and let the community test them as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] gengliangwang commented on a diff in pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on code in PR #400:
URL: https://github.com/apache/spark-website/pull/400#discussion_r906444239


##########
site/sitemap.xml:
##########
@@ -941,27 +941,27 @@
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/graphx/</loc>
+  <loc>https://spark.apache.org/news/</loc>

Review Comment:
   +1 @srowen. The changes on this file seem not necessary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] gengliangwang commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1166190151

   > Maybe, unlike maven repos though we don't have a staging location set up, I think we could ask ASF Infra to make us a staging location?
   
   We can publish RC images with a different tag, e.g. v3.4.0-rc1.
   After release, the images can be deleted.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] holdenk commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
holdenk commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1166189409

   Maybe, unlike maven repos though we don't have a staging location set up, I think we could ask ASF Infra to make us a staging location?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] gengliangwang commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1166208036

   > Sounds good. Want me to add the instructions to the RC step too then?
   
   Sure 👍 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] srowen commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
srowen commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1174003051

   Ping @holdenk 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] gengliangwang commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1166188788

   @holdenk I followed the steps and it works!
   I have built docker images on https://hub.docker.com/u/gengliangwang
   If @MaxGekk doesn't have permission to publish it, I can do it for him this time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] gengliangwang commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1166192744

   FYI I just published docker images for Spark 3.3. release
   https://hub.docker.com/r/apache/spark
   https://hub.docker.com/r/apache/spark-py
   https://hub.docker.com/r/apache/spark-r
   
   I will do send an email to the dev/user list if no issues found during the weekend
   cc @holdenk 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] tgravescs commented on a diff in pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
tgravescs commented on code in PR #400:
URL: https://github.com/apache/spark-website/pull/400#discussion_r907930914


##########
site/release-process.html:
##########
@@ -497,6 +499,14 @@ <h4>Update the Spark website</h4>
 $ git log v1.1.1 --grep "$expr" --shortstat --oneline | grep -B 1 -e "[3-9][0-9][0-9] insert" -e "[1-9][1-9][1-9][1-9] insert" | grep SPARK &gt; large-patches.txt
 </code></pre></div></div>
 
+<h4>Create and upload Spark Docker Images</h4>
+
+<p>The Spark docker images are created using the <code class="language-plaintext highlighter-rouge">./bin/docker-image-tool.sh</code> that is included in the release artifacts.</p>
+
+<p>You should install <code class="language-plaintext highlighter-rouge">docker buildx</code> so that you can cross-compile for multiple archs as ARM is becoming increasing popular. If you have access to both an ARM and an x86 machine you should set up a <a href="https://scalingpythonml.com/2020/12/11/some-sharp-corners-with-docker-buildx.html">remote builder as described here</a>, but if you only have one <a href="https://docs.docker.com/buildx/working-with-buildx/">docker buildx with QEMU works fine as we don&#8217;t use cgo</a>.</p>
+
+<p>Once you have your cross-platform docker build environment setup, extract the build artifact (e.g. <code class="language-plaintext highlighter-rouge">tar -xvf spark-3.3.0-bin-hadoop3.tgz</code>), go into the director (e.g. <code class="language-plaintext highlighter-rouge">cd spark-3.3.0-bin-hadoop3</code>) and build the containers and publish them to the Spark dockerhub (e.g. <code class="language-plaintext highlighter-rouge">./bin/docker-image-tool.sh -r docker.io/apache -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile -t v3.3.0 -X -b java_image_tag=11-jre-slim build</code>)</p>

Review Comment:
   spelling directory



##########
release-process.md:
##########
@@ -387,6 +390,16 @@ $ git shortlog v1.1.1 --grep "$EXPR" > contrib.txt
 $ git log v1.1.1 --grep "$expr" --shortstat --oneline | grep -B 1 -e "[3-9][0-9][0-9] insert" -e "[1-9][1-9][1-9][1-9] insert" | grep SPARK > large-patches.txt
 ```
 
+<h4>Create and upload Spark Docker Images</h4>
+
+The Spark docker images are created using the `./bin/docker-image-tool.sh` that is included in the release artifacts.
+
+
+You should install `docker buildx` so that you can cross-compile for multiple archs as ARM is becoming increasing popular. If you have access to both an ARM and an x86 machine you should set up a [remote builder as described here](https://scalingpythonml.com/2020/12/11/some-sharp-corners-with-docker-buildx.html), but if you only have one [docker buildx with QEMU works fine as we don't use cgo](https://docs.docker.com/buildx/working-with-buildx/).
+
+
+Once you have your cross-platform docker build environment setup, extract the build artifact (e.g. `tar -xvf spark-3.3.0-bin-hadoop3.tgz`), go into the director (e.g. `cd spark-3.3.0-bin-hadoop3`) and build the containers and publish them to the Spark dockerhub (e.g. `./bin/docker-image-tool.sh -r docker.io/apache -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile -t v3.3.0 -X -b java_image_tag=11-jre-slim build`)

Review Comment:
   spelling of directory



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] holdenk commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
holdenk commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1166205843

   Sounds good. Want me to add the instructions to the RC step too then?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] gengliangwang commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1188495099

   Merging to asf-site


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] dongjoon-hyun commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1189305437

   Thank you, @holdenk and all. According to this guideline, Apache Spark 3.2.2 images are also published too.
   
   <img width="711" alt="Screen Shot 2022-07-19 at 9 27 31 AM" src="https://user-images.githubusercontent.com/9700541/179801611-3e2ab69f-820a-4ea1-8800-e9253a26709e.png">
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] gengliangwang merged pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
gengliangwang merged PR #400:
URL: https://github.com/apache/spark-website/pull/400


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] holdenk commented on pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
holdenk commented on PR #400:
URL: https://github.com/apache/spark-website/pull/400#issuecomment-1176618299

   ping @MaxGekk & @tgravescs @gengliangwang since y'all had comments on the first draft, this one looking ok?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] srowen commented on a diff in pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
srowen commented on code in PR #400:
URL: https://github.com/apache/spark-website/pull/400#discussion_r904998278


##########
site/sitemap.xml:
##########
@@ -941,27 +941,27 @@
   <changefreq>weekly</changefreq>
 </url>
 <url>
-  <loc>https://spark.apache.org/graphx/</loc>
+  <loc>https://spark.apache.org/news/</loc>

Review Comment:
   @holdenk just checking if you saw this - do we want to revert or is this the right order now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org

[GitHub] [spark-website] tgravescs commented on a diff in pull request #400: [SPARK-39512] Document docker image release steps

Posted by GitBox <gi...@apache.org>.
tgravescs commented on code in PR #400:
URL: https://github.com/apache/spark-website/pull/400#discussion_r907930177


##########
release-process.md:
##########
@@ -35,6 +35,9 @@ If you are a new Release Manager, you can read up on the process from the follow
 - gpg for signing https://www.apache.org/dev/openpgp.html
 - svn https://www.apache.org/dev/version-control.html#https-svn
 
+
+You should also get access to the ASF Dockerhub, this is not required for the release candidates but can take some time. You can get access by filing a INFRA JIRA ticket.

Review Comment:
   do we want to add a link to infra jira, or instructions, or example ticket someone already did this with to make this a bit easier on someone new?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org