You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "tcodehuber (via GitHub)" <gi...@apache.org> on 2024/05/08 06:00:13 UTC
[PR] [AMORO-2815] Adjust docker images of apache amoro project [amoro]
tcodehuber opened a new pull request, #2818:
URL: https://github.com/apache/amoro/pull/2818
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://amoro.apache.org/how-to-contribute/
2. If the PR is related to an issue in https://github.com/apache/amoro/issues, add '[AMORO-XXXX]' in your PR title, e.g., '[AMORO-XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][AMORO-XXXX] Your PR title ...'.
-->
## Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about its use case.
2. If you fix a bug, you can clarify why it is a bug.
3. Use Fix/Resolve/Close #{ISSUE_NUMBER} to link this PR to its related issue
-->
Close #2815.
## Brief change log
<!--
Clearly describe the changes made in modules, classes, methods, etc.
-->
-
## How was this patch tested?
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] Run test locally before making a pull request
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-2815] Adjust docker images of apache amoro project [amoro]
Posted by "tcodehuber (via GitHub)" <gi...@apache.org>.
tcodehuber commented on code in PR #2818:
URL: https://github.com/apache/amoro/pull/2818#discussion_r1594917429
##########
docker/build.sh:
##########
@@ -44,23 +42,18 @@ Usage: $0 [options] [image]
Build for Amoro demo docker images.
Images:
- quickdemo Build Amoro QuickStart Image, for run flink ingestion job in quick-demo http://amoro.apache.org/quick-demo/
- namenode Build a hadoop namenode container for quick start demo.
- datanode Build a hadoop datanode container for quick start demo.
optimizer-flink Build official Amoro optimizer deployed with flink engine for production environments.
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-2815] Adjust docker images of apache amoro project [amoro]
Posted by "tcodehuber (via GitHub)" <gi...@apache.org>.
tcodehuber commented on code in PR #2818:
URL: https://github.com/apache/amoro/pull/2818#discussion_r1594917115
##########
amoro-ams/dist/src/main/arctic-bin/conf/config.yaml:
##########
@@ -142,7 +142,7 @@ containers:
# container-impl: com.netease.arctic.server.manager.KubernetesOptimizerContainer
# properties:
# kube-config-path: ～/.kube/config
-# image: arctic163/amoro:0.6
+# image: apache/amoro:0.6
Review Comment:
how about using {tag}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-2815] Adjust docker images of apache amoro project [amoro]
Posted by "tcodehuber (via GitHub)" <gi...@apache.org>.
tcodehuber commented on code in PR #2818:
URL: https://github.com/apache/amoro/pull/2818#discussion_r1594917115
##########
amoro-ams/dist/src/main/arctic-bin/conf/config.yaml:
##########
@@ -142,7 +142,7 @@ containers:
# container-impl: com.netease.arctic.server.manager.KubernetesOptimizerContainer
# properties:
# kube-config-path: ～/.kube/config
-# image: arctic163/amoro:0.6
+# image: apache/amoro:0.6
Review Comment:
how about using {tag}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-2815] Adjust docker images of apache amoro project [amoro]
Posted by "tcodehuber (via GitHub)" <gi...@apache.org>.
tcodehuber commented on code in PR #2818:
URL: https://github.com/apache/amoro/pull/2818#discussion_r1594913346
##########
charts/amoro/values.yaml:
##########
@@ -254,7 +254,7 @@ optimizer:
properties:
namespace: "default"
kube-config-path: "～/.kube/config"
- image: "arctic163/amoro:0.6"
+ image: "apache/amoro:0.6"
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-2815] Adjust docker images of apache amoro project [amoro]
Posted by "baiyangtx (via GitHub)" <gi...@apache.org>.
baiyangtx commented on code in PR #2818:
URL: https://github.com/apache/amoro/pull/2818#discussion_r1594869153
##########
charts/amoro/values.yaml:
##########
@@ -254,7 +254,7 @@ optimizer:
properties:
namespace: "default"
kube-config-path: "～/.kube/config"
- image: "arctic163/amoro:0.6"
+ image: "apache/amoro:0.6"
Review Comment:
We shouldn't hardcore a version in the default values.yml. I will fix this in the other PR.
##########
docker/demo-cluster.sh:
##########
@@ -1,186 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-# *
-# http://www.apache.org/licenses/LICENSE-2.0
-# *
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-# default to use the last built image when no specified tag given
-AMORO_TAG=latest
-
-
-CURRENT_DIR="$( cd "$(dirname "$0")" ; pwd -P )"
-
-
-DOCKER_COMPOSE="${CURRENT_DIR}/docker-compose.yml"
-HADOOP_CONF="${CURRENT_DIR}/hadoop-config"
-VOLUMES_DIR="${CURRENT_DIR}/volumes"
-
-function usage() {
- cat <<EOF
-Usage: $0 [options] [command]
-Build for Amoro demo docker images.
-
-Commands:
- start Setup demo cluster
- stop Stop demo cluster and remove containers, volume data will be kept.
- clean clean volume data.
-
-Options:
- -v --version Setup Amoro image version. default is ${AMORO_TAG}
-
-EOF
-}
Review Comment:
I think we shoud keep demo-cluster.sh but using apache/amoro and minio to create demo cluster.
##########
docker/demo-cluster.sh:
##########
@@ -1,186 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-# *
-# http://www.apache.org/licenses/LICENSE-2.0
-# *
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-# default to use the last built image when no specified tag given
-AMORO_TAG=latest
-
-
-CURRENT_DIR="$( cd "$(dirname "$0")" ; pwd -P )"
-
-
-DOCKER_COMPOSE="${CURRENT_DIR}/docker-compose.yml"
-HADOOP_CONF="${CURRENT_DIR}/hadoop-config"
-VOLUMES_DIR="${CURRENT_DIR}/volumes"
-
-function usage() {
- cat <<EOF
-Usage: $0 [options] [command]
-Build for Amoro demo docker images.
-
-Commands:
- start Setup demo cluster
- stop Stop demo cluster and remove containers, volume data will be kept.
- clean clean volume data.
-
-Options:
- -v --version Setup Amoro image version. default is ${AMORO_TAG}
-
-EOF
-}
Review Comment:
@zhoujinsong WDYT? or we can manage demo-cluster.sh in amoro-site project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-2815] Adjust docker images of apache amoro project [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on code in PR #2818:
URL: https://github.com/apache/amoro/pull/2818#discussion_r1594875167
##########
docker/README.md:
##########
@@ -24,18 +24,15 @@ You can control which image is built by the follow parameters :
- amoro
- optimizer-flink
- optimizer-spark
-- quickdemo
-- namenode
-- datanode
e.g.
```shell
./build.sh amoro
```
-- NOTICE: The amoro image, quickdemo image and optimizer-flink image required the project had been packaged.
-so run `mvn package -pl '!trino'` before build amoro, quickdemo or optimizer-flink image.
+- NOTICE: The amoro image, amoro-spark-optimizer image and amoro-flink-optimizer image required the project had been packaged.
+so run `mvn package -pl '!trino'` before build amoro, amoro-spark-optimizer or amoro-flink-optimizer image.
Review Comment:
```suggestion
So run `mvn package -pl '!amoro-mixed-fomrat/amoro-mixed-format-trino'` before building amoro, amoro-spark-optimizer, or amoro-flink-optimizer image.
```
##########
docker/build.sh:
##########
@@ -44,23 +42,18 @@ Usage: $0 [options] [image]
Build for Amoro demo docker images.
Images:
- quickdemo Build Amoro QuickStart Image, for run flink ingestion job in quick-demo http://amoro.apache.org/quick-demo/
- namenode Build a hadoop namenode container for quick start demo.
- datanode Build a hadoop datanode container for quick start demo.
optimizer-flink Build official Amoro optimizer deployed with flink engine for production environments.
Review Comment:
How about changing the name to `amoro-flink-optimizer`/`amoro-spark-optimizer` here?
##########
charts/amoro/values.yaml:
##########
@@ -254,7 +254,7 @@ optimizer:
properties:
namespace: "default"
kube-config-path: "～/.kube/config"
- image: "arctic163/amoro:0.6"
+ image: "apache/amoro:0.6"
Review Comment:
I'm concerned about whether specifying version 0.6 here is appropriate.
##########
amoro-ams/dist/src/main/arctic-bin/conf/config.yaml:
##########
@@ -142,7 +142,7 @@ containers:
# container-impl: com.netease.arctic.server.manager.KubernetesOptimizerContainer
# properties:
# kube-config-path: ～/.kube/config
-# image: arctic163/amoro:0.6
+# image: apache/amoro:0.6
Review Comment:
Maybe replace `0.6` with `{version}`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-2815] Adjust docker images of apache amoro project [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on code in PR #2818:
URL: https://github.com/apache/amoro/pull/2818#discussion_r1594878485
##########
docker/demo-cluster.sh:
##########
@@ -1,186 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-# *
-# http://www.apache.org/licenses/LICENSE-2.0
-# *
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-# default to use the last built image when no specified tag given
-AMORO_TAG=latest
-
-
-CURRENT_DIR="$( cd "$(dirname "$0")" ; pwd -P )"
-
-
-DOCKER_COMPOSE="${CURRENT_DIR}/docker-compose.yml"
-HADOOP_CONF="${CURRENT_DIR}/hadoop-config"
-VOLUMES_DIR="${CURRENT_DIR}/volumes"
-
-function usage() {
- cat <<EOF
-Usage: $0 [options] [command]
-Build for Amoro demo docker images.
-
-Commands:
- start Setup demo cluster
- stop Stop demo cluster and remove containers, volume data will be kept.
- clean clean volume data.
-
-Options:
- -v --version Setup Amoro image version. default is ${AMORO_TAG}
-
-EOF
-}
Review Comment:
I am not sure if we still need a shell script to create the docker-compose file or if just a compose example file is okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-2815] Adjust docker images of apache amoro project [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong merged PR #2818:
URL: https://github.com/apache/amoro/pull/2818
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-2815] Adjust docker images of apache amoro project [amoro]
Posted by "tcodehuber (via GitHub)" <gi...@apache.org>.
tcodehuber commented on code in PR #2818:
URL: https://github.com/apache/amoro/pull/2818#discussion_r1594914477
##########
docker/demo-cluster.sh:
##########
@@ -1,186 +0,0 @@
-#!/usr/bin/env bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-# *
-# http://www.apache.org/licenses/LICENSE-2.0
-# *
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-# default to use the last built image when no specified tag given
-AMORO_TAG=latest
-
-
-CURRENT_DIR="$( cd "$(dirname "$0")" ; pwd -P )"
-
-
-DOCKER_COMPOSE="${CURRENT_DIR}/docker-compose.yml"
-HADOOP_CONF="${CURRENT_DIR}/hadoop-config"
-VOLUMES_DIR="${CURRENT_DIR}/volumes"
-
-function usage() {
- cat <<EOF
-Usage: $0 [options] [command]
-Build for Amoro demo docker images.
-
-Commands:
- start Setup demo cluster
- stop Stop demo cluster and remove containers, volume data will be kept.
- clean clean volume data.
-
-Options:
- -v --version Setup Amoro image version. default is ${AMORO_TAG}
-
-EOF
-}
Review Comment:
I will rebuild the demo process in another PR to fix #2816, importing a docker-compose.yml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org