You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/08 07:04:01 UTC
[GitHub] [spark-docker] Yikun opened a new pull request, #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
Yikun opened a new pull request, #23:
URL: https://github.com/apache/spark-docker/pull/23
### What changes were proposed in this pull request?
The publish step will include 3 steps:
1. First build the local image.
2. Pass related test using image of first step.
3. After pass all test, will publish to `ghcr` (This might help RC test) or `dockerhub`
There would be 30-40 mins to publish all images.
Add "Publish" workflow to help release apache/spark image.
### Why are the changes needed?
One click to create the `apche/spark` image.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
1. Set default branch in my fork repo
2. Run workflow manually, https://github.com/Yikun/spark-docker/actions/workflows/publish.yml?query=is%3Asuccess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark-docker] Yikun commented on pull request #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #23:
URL: https://github.com/apache/spark-docker/pull/23#issuecomment-1309912799
cc @HyukjinKwon @zhengruifeng ready to go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark-docker] Yikun commented on pull request #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #23:
URL: https://github.com/apache/spark-docker/pull/23#issuecomment-1315319292
@HyukjinKwon @martin-g Thanks! Merge to master. Looks like no more feedback about publishing. Maybe let's publish after docker official image review completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark-docker] Yikun commented on pull request #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #23:
URL: https://github.com/apache/spark-docker/pull/23#issuecomment-1309912414
Publish test to ghcr (ghcr/yikun): https://github.com/Yikun/spark-docker/actions/runs/3434472592/jobs/5725812762
Publish test to dockerhub (yikunkero): https://github.com/Yikun/spark-docker/actions/runs/3434473749/jobs/5725814930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark-docker] Yikun commented on a diff in pull request #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
Posted by GitBox <gi...@apache.org>.
Yikun commented on code in PR #23:
URL: https://github.com/apache/spark-docker/pull/23#discussion_r1016246024
##########
.github/workflows/publish.yml:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Publish"
+
+on:
+ workflow_dispatch:
+ inputs:
+ spark:
+ description: 'The Spark version of Spark image.'
+ required: true
+ default: '3.3.0'
+ type: choice
+ options:
+ - 3.3.0
+ - 3.3.1
+ scala:
+ description: 'The Scala version of Spark image.'
+ required: true
+ type: string
Review Comment:
Yep, this is not just simple add a option, we need add 2.13 support separately in future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark-docker] martin-g commented on a diff in pull request #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
Posted by GitBox <gi...@apache.org>.
martin-g commented on code in PR #23:
URL: https://github.com/apache/spark-docker/pull/23#discussion_r1016218102
##########
.github/workflows/publish.yml:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Publish"
+
+on:
+ workflow_dispatch:
+ inputs:
+ spark:
+ description: 'The Spark version of Spark image.'
+ required: true
+ default: '3.3.0'
+ type: choice
+ options:
+ - 3.3.0
+ - 3.3.1
+ scala:
+ description: 'The Scala version of Spark image.'
+ required: true
+ type: string
Review Comment:
why not `choice` with options `'2.12'` and `'2.13'` ?
##########
.github/workflows/publish.yml:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Publish"
+
+on:
+ workflow_dispatch:
+ inputs:
+ spark:
+ description: 'The Spark version of Spark image.'
+ required: true
+ default: '3.3.0'
+ type: choice
+ options:
+ - 3.3.0
+ - 3.3.1
+ scala:
+ description: 'The Scala version of Spark image.'
+ required: true
+ type: string
+ default: 2.12
+ java:
+ description: 'The Java version of Spark image.'
+ required: true
+ type: string
+ default: 11
+ publish:
+ description: 'Publish the image or not.'
+ default: '3.3.0'
Review Comment:
it should be `true` or `false`
##########
.github/workflows/publish.yml:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Publish"
+
+on:
+ workflow_dispatch:
+ inputs:
+ spark:
+ description: 'The Spark version of Spark image.'
+ required: true
+ default: '3.3.0'
+ type: choice
+ options:
+ - 3.3.0
+ - 3.3.1
+ scala:
+ description: 'The Scala version of Spark image.'
+ required: true
+ type: string
+ default: 2.12
Review Comment:
nit: wrap the value in single quotes as L28 ?!
##########
versions.json:
##########
@@ -0,0 +1,35 @@
+{
+ "versions": [
Review Comment:
here I see only images with Scala 2.12, so my earlier comment about using a `choice` for 2.12/2.13 maybe is not needed
##########
tools/manifest.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from argparse import ArgumentParser
+import json
+from statistics import mode
+
+
+def parse_opts():
+ parser = ArgumentParser(prog="manifest.py")
+
+ parser.add_argument(
+ dest="mode",
+ choices=["tags"],
+ type=str,
+ help="The path to specific dockerfile",
Review Comment:
copy/paste error ?!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark-docker] Yikun commented on pull request #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #23:
URL: https://github.com/apache/spark-docker/pull/23#issuecomment-1309832054
After some considerations, we'd better only expose the `Spark version` and `Publish repo`, and move inputs.scala/java in to matrix, because:
1. It's simpler, the release manager just need to select the version to publish and ignore the java/scala verison, it should be integrated in developer process.
2. we can simple to use matrix to generate all jobs, such as if we support the multiple java version in future, the workflow looks like:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark-docker] Yikun closed pull request #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
Posted by GitBox <gi...@apache.org>.
Yikun closed pull request #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
URL: https://github.com/apache/spark-docker/pull/23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark-docker] Yikun commented on a diff in pull request #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
Posted by GitBox <gi...@apache.org>.
Yikun commented on code in PR #23:
URL: https://github.com/apache/spark-docker/pull/23#discussion_r1016252899
##########
versions.json:
##########
@@ -0,0 +1,35 @@
+{
+ "versions": [
Review Comment:
Yep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark-docker] Yikun commented on pull request #23: [SPARK-40519] Add "Publish" workflow to help release apache/spark image
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #23:
URL: https://github.com/apache/spark-docker/pull/23#issuecomment-1309916859
Due to slow progress of docker official image, we might want to publish `apache/spark` first (3.3.0/3.3.1), if no more objection, I will merge this today (The ability to publish with one click).
And I will also send a mail list later to see feedback about publishing `apache/spark` image first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org