You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/08/24 16:46:48 UTC
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #5911: Add compatibility verifier scripts (#4854)
mcvsubbu commented on a change in pull request #5911:
URL: https://github.com/apache/incubator-pinot/pull/5911#discussion_r475728734
##########
File path: compatibility-verifier/build-release.sh
##########
@@ -0,0 +1,59 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A helper script that builds Pinot given a specific commit hash and target directory
+# This script is intended to be called stand-alone or from comp-verifier.sh
+# command syntax: ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2
Review comment:
Let us use this script to build one release, and invoke it as many times as needed
##########
File path: compatibility-verifier/build-release.sh
##########
@@ -0,0 +1,59 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A helper script that builds Pinot given a specific commit hash and target directory
+# This script is intended to be called stand-alone or from comp-verifier.sh
+# command syntax: ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2
+
+# verify correct usage of this script
+if [[ $# -ne 2 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./build-release.sh commit_hash target_dir \n"
Review comment:
Introduce a function called usage() and print the usage in it. We can call that function from multiple points as necessary
##########
File path: compatibility-verifier/build-release.sh
##########
@@ -0,0 +1,59 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A helper script that builds Pinot given a specific commit hash and target directory
+# This script is intended to be called stand-alone or from comp-verifier.sh
+# command syntax: ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2
+
+# verify correct usage of this script
+if [[ $# -ne 2 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./build-release.sh commit_hash target_dir \n"
+ exit 1
+fi
+
+# get arguments
+commit_hash=$1
+target_dir=$2
+
+# check if directory already exists and checkout
Review comment:
exit if the directory exists. We can add a -f argument later or force
##########
File path: compatibility-verifier/comp-verifier.sh
##########
@@ -0,0 +1,145 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A script that does rolling upgrade of Pinot components
+# from one version to the other given 2 commit hashes. It first builds
Review comment:
Mention clearly that the script assumes the first hash is the earlier one and the second is the later one. Change your variable names to indicate so
##########
File path: compatibility-verifier/comp-verifier.sh
##########
@@ -0,0 +1,145 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A script that does rolling upgrade of Pinot components
+# from one version to the other given 2 commit hashes. It first builds
+# Pinot in the 2 given directories and then upgrades in the following order:
+# Controller -> Broker -> Server
+
+# verify correct usage of this script
+if [[ $# -ne 4 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2 \n"
+ exit 1
+fi
+
+# get arguments
+commitHash1=$1
+target_dir_1=$2
+commitHash2=$3
+target_dir_2=$4
+pinot_version_1="0.5.0"
+pinot_version_2="0.5.0"
+
+read -rep $'\n' -p "What is the Pinot version for first commitHash? [default: 0.5.0] " -r
+if [[ ! $REPLY == "" ]]; then
+ pinot_version_1=$REPLY
+fi
+read -rep $'\n' -p "What is the Pinot version for second commitHash? [default: 0.5.0] " -r
+if [[ ! $REPLY == "" ]]; then
+ pinot_version_2=$REPLY
+fi
+
+# Building targets
+read -rep $'\n' -p "Do you want to build the first target? [default: no] " -n 1 -r
+if [[ $REPLY =~ ^[Yy]$ ]]; then
+ printf "Building the first target ... \n"
+ ./build-release.sh "$commitHash1" "$target_dir_1"
+fi
+
+read -rep $'\n' -p "Do you want to build the second target? [default: no] " -n 1 -r
+if [[ $REPLY =~ ^[Yy]$ ]]; then
+ printf "Building the second target ... \n"
+ ./build-release.sh "$commitHash2" "$target_dir_2"
+fi
+
+if [[ $(lsof -t -i:7001 -s TCP:LISTEN) || $(lsof -t -i:8001 -sTCP:LISTEN) || $(lsof -t -i:9001 -sTCP:LISTEN) ||
Review comment:
Not sure if lsof returns all ports even if you don't have permissions
##########
File path: compatibility-verifier/control-service.sh
##########
@@ -0,0 +1,104 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A helper script that starts/stops given component for a given build of Pinot
+# This script is intended to be called stand-alone or from comp-verifier.sh
+# command syntax: ./control-service.sh command component target_dir pinot_version
Review comment:
You don't need pinot version
##########
File path: compatibility-verifier/build-release.sh
##########
@@ -0,0 +1,59 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A helper script that builds Pinot given a specific commit hash and target directory
+# This script is intended to be called stand-alone or from comp-verifier.sh
+# command syntax: ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2
+
+# verify correct usage of this script
+if [[ $# -ne 2 ]]; then
Review comment:
Please use '[' instead of '[['. Won't repeat for other instances further down in this file and other files
##########
File path: compatibility-verifier/build-release.sh
##########
@@ -0,0 +1,59 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A helper script that builds Pinot given a specific commit hash and target directory
+# This script is intended to be called stand-alone or from comp-verifier.sh
+# command syntax: ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2
+
+# verify correct usage of this script
+if [[ $# -ne 2 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./build-release.sh commit_hash target_dir \n"
Review comment:
```suggestion
echo "Usage:$0 commit_hash target_dir "
```
##########
File path: compatibility-verifier/comp-verifier.sh
##########
@@ -0,0 +1,145 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A script that does rolling upgrade of Pinot components
+# from one version to the other given 2 commit hashes. It first builds
+# Pinot in the 2 given directories and then upgrades in the following order:
+# Controller -> Broker -> Server
+
+# verify correct usage of this script
+if [[ $# -ne 4 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2 \n"
Review comment:
```suggestion
echo "Usage: $0 earlierCommitHash laterCommitHash workingDir"
```
##########
File path: compatibility-verifier/build-release.sh
##########
@@ -0,0 +1,59 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A helper script that builds Pinot given a specific commit hash and target directory
+# This script is intended to be called stand-alone or from comp-verifier.sh
+# command syntax: ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2
+
+# verify correct usage of this script
+if [[ $# -ne 2 ]]; then
+ printf "You used %s arguments \n" "$#"
Review comment:
Remove this line
##########
File path: compatibility-verifier/build-release.sh
##########
@@ -0,0 +1,59 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A helper script that builds Pinot given a specific commit hash and target directory
+# This script is intended to be called stand-alone or from comp-verifier.sh
+# command syntax: ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2
+
+# verify correct usage of this script
+if [[ $# -ne 2 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./build-release.sh commit_hash target_dir \n"
+ exit 1
+fi
+
+# get arguments
+commit_hash=$1
+target_dir=$2
+
+# check if directory already exists and checkout
+# using the given commit
+if [[ ! -d $target_dir ]]; then
+ mkdir -p "$target_dir"
Review comment:
You will need to check the exit status of mkdir in case there were permission issues
##########
File path: compatibility-verifier/comp-verifier.sh
##########
@@ -0,0 +1,145 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A script that does rolling upgrade of Pinot components
+# from one version to the other given 2 commit hashes. It first builds
+# Pinot in the 2 given directories and then upgrades in the following order:
+# Controller -> Broker -> Server
+
+# verify correct usage of this script
+if [[ $# -ne 4 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2 \n"
+ exit 1
+fi
+
+# get arguments
+commitHash1=$1
+target_dir_1=$2
+commitHash2=$3
+target_dir_2=$4
+pinot_version_1="0.5.0"
+pinot_version_2="0.5.0"
+
+read -rep $'\n' -p "What is the Pinot version for first commitHash? [default: 0.5.0] " -r
Review comment:
Why read pinot versions? We have the commit hash, pick up whatever version or hash that is.
##########
File path: compatibility-verifier/comp-verifier.sh
##########
@@ -0,0 +1,145 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A script that does rolling upgrade of Pinot components
+# from one version to the other given 2 commit hashes. It first builds
+# Pinot in the 2 given directories and then upgrades in the following order:
+# Controller -> Broker -> Server
+
+# verify correct usage of this script
+if [[ $# -ne 4 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2 \n"
+ exit 1
+fi
+
+# get arguments
+commitHash1=$1
+target_dir_1=$2
+commitHash2=$3
+target_dir_2=$4
+pinot_version_1="0.5.0"
+pinot_version_2="0.5.0"
+
+read -rep $'\n' -p "What is the Pinot version for first commitHash? [default: 0.5.0] " -r
+if [[ ! $REPLY == "" ]]; then
+ pinot_version_1=$REPLY
+fi
+read -rep $'\n' -p "What is the Pinot version for second commitHash? [default: 0.5.0] " -r
+if [[ ! $REPLY == "" ]]; then
+ pinot_version_2=$REPLY
+fi
+
+# Building targets
+read -rep $'\n' -p "Do you want to build the first target? [default: no] " -n 1 -r
+if [[ $REPLY =~ ^[Yy]$ ]]; then
+ printf "Building the first target ... \n"
+ ./build-release.sh "$commitHash1" "$target_dir_1"
+fi
+
+read -rep $'\n' -p "Do you want to build the second target? [default: no] " -n 1 -r
+if [[ $REPLY =~ ^[Yy]$ ]]; then
+ printf "Building the second target ... \n"
+ ./build-release.sh "$commitHash2" "$target_dir_2"
+fi
+
+if [[ $(lsof -t -i:7001 -s TCP:LISTEN) || $(lsof -t -i:8001 -sTCP:LISTEN) || $(lsof -t -i:9001 -sTCP:LISTEN) ||
+ $(lsof -t -i:2181 -sTCP:LISTEN) || $(lsof -t -i:8011 -sTCP:LISTEN) ]]; then
+ read -rep $'\n' -p "The ports need to be open. Do you want to kill existing processes on these ports? " -n 1 -r
+ if [[ $REPLY =~ ^[Yy]$ ]]; then
+ ## Clean up the ports and kill any processes running on them
+ if [[ $(lsof -t -i:7001 -s TCP:LISTEN) ]]; then
+ kill -9 "$(lsof -t -i:7001 -s TCP:LISTEN)"
+ fi
+ if [[ $(lsof -t -i:8001 -s TCP:LISTEN) ]]; then
+ kill -9 "$(lsof -t -i:8001 -s TCP:LISTEN)"
+ fi
+ if [[ $(lsof -t -i:9001 -s TCP:LISTEN) ]]; then
+ kill -9 "$(lsof -t -i:9001 -s TCP:LISTEN)"
+ fi
+ if [[ $(lsof -t -i:2181 -s TCP:LISTEN) ]]; then
+ kill -9 "$(lsof -t -i:2181 -s TCP:LISTEN)"
+ fi
+ if [[ $(lsof -t -i:8011 -s TCP:LISTEN) ]]; then
+ kill -9 "$(lsof -t -i:8011 -s TCP:LISTEN)"
+ fi
+ fi
+fi
+
+read -rep $'\n' -p "Do you want to build the cluster with first commit? [default: no] " -n 1 -r
+if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+ exit 0;
+fi
+
+
+# Setup initial cluster (with commit in target_dir_1)
+
+printf "\n\nStarting Zookeeper ...\n"
Review comment:
You don't need these prompts.
##########
File path: compatibility-verifier/control-service.sh
##########
@@ -0,0 +1,104 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A helper script that starts/stops given component for a given build of Pinot
+# This script is intended to be called stand-alone or from comp-verifier.sh
+# command syntax: ./control-service.sh command component target_dir pinot_version
+
+# verify correct usage of this script
+if [[ $# -ne 4 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./control-service.sh command component target_dir pinot_version \n"
+ exit 1
+fi
+
+# get arguments
+command=$1
+component=$2
+target_dir=$3
+pinot_version=$4
+
+declare -a commands=("start" "stop")
+declare -a components=("controller" "broker" "server" "zookeeper")
+
+if [[ ! -d "$target_dir"/incubator-pinot ]]; then
+ printf "%s/incubator-pinot does not exist \n " "$target_dir"
+ exit 1
+fi
+
+# Handle invalid arguments
+if [[ ! " ${commands[*]} " =~ $command ]]; then
+ printf "%s is not a valid command. Command must be one of: start and stop\n" "$command"
+ printf "Usage: \n ./controlService.sh command component target_dir \n"
+ exit 1
+fi
+if [[ ! " ${components[*]} " =~ $component ]]; then
+ printf "Not a valid component. Needs to be one of: %s \n" "${components[*]}"
+ exit 1
+fi
+
+# Define ports for the test
+zkPort=2181
Review comment:
For now, just use the default ports on the service. Let us get a basic framework working. and then we can add the bells and whistles.
##########
File path: compatibility-verifier/comp-verifier.sh
##########
@@ -0,0 +1,145 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A script that does rolling upgrade of Pinot components
+# from one version to the other given 2 commit hashes. It first builds
+# Pinot in the 2 given directories and then upgrades in the following order:
+# Controller -> Broker -> Server
+
+# verify correct usage of this script
+if [[ $# -ne 4 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2 \n"
+ exit 1
+fi
+
+# get arguments
+commitHash1=$1
+target_dir_1=$2
+commitHash2=$3
+target_dir_2=$4
+pinot_version_1="0.5.0"
+pinot_version_2="0.5.0"
+
+read -rep $'\n' -p "What is the Pinot version for first commitHash? [default: 0.5.0] " -r
+if [[ ! $REPLY == "" ]]; then
+ pinot_version_1=$REPLY
+fi
+read -rep $'\n' -p "What is the Pinot version for second commitHash? [default: 0.5.0] " -r
+if [[ ! $REPLY == "" ]]; then
+ pinot_version_2=$REPLY
+fi
+
+# Building targets
+read -rep $'\n' -p "Do you want to build the first target? [default: no] " -n 1 -r
+if [[ $REPLY =~ ^[Yy]$ ]]; then
+ printf "Building the first target ... \n"
+ ./build-release.sh "$commitHash1" "$target_dir_1"
+fi
+
+read -rep $'\n' -p "Do you want to build the second target? [default: no] " -n 1 -r
+if [[ $REPLY =~ ^[Yy]$ ]]; then
+ printf "Building the second target ... \n"
+ ./build-release.sh "$commitHash2" "$target_dir_2"
+fi
+
+if [[ $(lsof -t -i:7001 -s TCP:LISTEN) || $(lsof -t -i:8001 -sTCP:LISTEN) || $(lsof -t -i:9001 -sTCP:LISTEN) ||
+ $(lsof -t -i:2181 -sTCP:LISTEN) || $(lsof -t -i:8011 -sTCP:LISTEN) ]]; then
+ read -rep $'\n' -p "The ports need to be open. Do you want to kill existing processes on these ports? " -n 1 -r
+ if [[ $REPLY =~ ^[Yy]$ ]]; then
+ ## Clean up the ports and kill any processes running on them
+ if [[ $(lsof -t -i:7001 -s TCP:LISTEN) ]]; then
+ kill -9 "$(lsof -t -i:7001 -s TCP:LISTEN)"
+ fi
+ if [[ $(lsof -t -i:8001 -s TCP:LISTEN) ]]; then
+ kill -9 "$(lsof -t -i:8001 -s TCP:LISTEN)"
+ fi
+ if [[ $(lsof -t -i:9001 -s TCP:LISTEN) ]]; then
+ kill -9 "$(lsof -t -i:9001 -s TCP:LISTEN)"
+ fi
+ if [[ $(lsof -t -i:2181 -s TCP:LISTEN) ]]; then
+ kill -9 "$(lsof -t -i:2181 -s TCP:LISTEN)"
+ fi
+ if [[ $(lsof -t -i:8011 -s TCP:LISTEN) ]]; then
+ kill -9 "$(lsof -t -i:8011 -s TCP:LISTEN)"
+ fi
+ fi
+fi
+
+read -rep $'\n' -p "Do you want to build the cluster with first commit? [default: no] " -n 1 -r
Review comment:
remove this prompt. The user has entered the command, and is therefore ready to build the cluster.
##########
File path: compatibility-verifier/comp-verifier.sh
##########
@@ -0,0 +1,145 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A script that does rolling upgrade of Pinot components
+# from one version to the other given 2 commit hashes. It first builds
+# Pinot in the 2 given directories and then upgrades in the following order:
+# Controller -> Broker -> Server
+
+# verify correct usage of this script
+if [[ $# -ne 4 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2 \n"
+ exit 1
+fi
+
+# get arguments
+commitHash1=$1
+target_dir_1=$2
+commitHash2=$3
+target_dir_2=$4
+pinot_version_1="0.5.0"
+pinot_version_2="0.5.0"
+
+read -rep $'\n' -p "What is the Pinot version for first commitHash? [default: 0.5.0] " -r
+if [[ ! $REPLY == "" ]]; then
+ pinot_version_1=$REPLY
+fi
+read -rep $'\n' -p "What is the Pinot version for second commitHash? [default: 0.5.0] " -r
+if [[ ! $REPLY == "" ]]; then
+ pinot_version_2=$REPLY
+fi
+
+# Building targets
+read -rep $'\n' -p "Do you want to build the first target? [default: no] " -n 1 -r
+if [[ $REPLY =~ ^[Yy]$ ]]; then
+ printf "Building the first target ... \n"
+ ./build-release.sh "$commitHash1" "$target_dir_1"
+fi
+
+read -rep $'\n' -p "Do you want to build the second target? [default: no] " -n 1 -r
+if [[ $REPLY =~ ^[Yy]$ ]]; then
+ printf "Building the second target ... \n"
+ ./build-release.sh "$commitHash2" "$target_dir_2"
+fi
+
+if [[ $(lsof -t -i:7001 -s TCP:LISTEN) || $(lsof -t -i:8001 -sTCP:LISTEN) || $(lsof -t -i:9001 -sTCP:LISTEN) ||
Review comment:
Do not kill existing processes. If the ports are not open, exit with an error saying so
We will add additional a config file for each component to fine-tune things. For now,
default configs and ports are good enough to get things going.
##########
File path: compatibility-verifier/comp-verifier.sh
##########
@@ -0,0 +1,145 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A script that does rolling upgrade of Pinot components
+# from one version to the other given 2 commit hashes. It first builds
+# Pinot in the 2 given directories and then upgrades in the following order:
+# Controller -> Broker -> Server
+
+# verify correct usage of this script
+if [[ $# -ne 4 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./comp-verifier.sh commitHash1 target_dir_1 commitHash2 target_dir_2 \n"
Review comment:
Take a working dir as argument and mkdir two subdirs in it, one for each commit hash.
##########
File path: compatibility-verifier/control-service.sh
##########
@@ -0,0 +1,104 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A helper script that starts/stops given component for a given build of Pinot
+# This script is intended to be called stand-alone or from comp-verifier.sh
+# command syntax: ./control-service.sh command component target_dir pinot_version
+
+# verify correct usage of this script
+if [[ $# -ne 4 ]]; then
+ printf "You used %s arguments \n" "$#"
+ printf "Usage: \n ./control-service.sh command component target_dir pinot_version \n"
+ exit 1
+fi
+
+# get arguments
+command=$1
+component=$2
+target_dir=$3
+pinot_version=$4
+
+declare -a commands=("start" "stop")
+declare -a components=("controller" "broker" "server" "zookeeper")
+
+if [[ ! -d "$target_dir"/incubator-pinot ]]; then
+ printf "%s/incubator-pinot does not exist \n " "$target_dir"
+ exit 1
+fi
+
+# Handle invalid arguments
+if [[ ! " ${commands[*]} " =~ $command ]]; then
+ printf "%s is not a valid command. Command must be one of: start and stop\n" "$command"
+ printf "Usage: \n ./controlService.sh command component target_dir \n"
+ exit 1
+fi
+if [[ ! " ${components[*]} " =~ $component ]]; then
+ printf "Not a valid component. Needs to be one of: %s \n" "${components[*]}"
+ exit 1
+fi
+
+# Define ports for the test
+zkPort=2181
+localhost=2181
+controllerPort=9001
+brokerPort=7001
+serverPort=8001
+serverAdminPort=8011
+
+if [[ "$command" == "start" ]]; then
+ # Navigate to the directory containing the scripts
+ if [[ $pinot_version == "0.3.0" ]]; then
+ pushd "$target_dir"/incubator-pinot/pinot-distribution/target/apache-pinot-incubating-"$pinot_version"-SNAPSHOT-bin/apache-pinot-incubating-"$pinot_version"-SNAPSHOT-bin/bin || exit 1
+ elif [[ $pinot_version == "0.4.0" ]]; then
+ pushd "$target_dir"/incubator-pinot/pinot-distribution/target/apache-pinot-incubating-"$pinot_version"-SNAPSHOT-bin/apache-pinot-incubating-"$pinot_version"-SNAPSHOT-bin/bin || exit 1
+ else
+ pushd "$target_dir"/incubator-pinot/pinot-distribution/target/apache-pinot-incubating-"$pinot_version"-SNAPSHOT-bin/apache-pinot-incubating-"$pinot_version"-SNAPSHOT-bin/bin || exit 1
+ fi
+
+ # Start the desired component
+ # Upon start, save the pid of the process for a component into a file in /tmp/{component}.pid, which is then used to stop it
+ if [[ "$component" == "controller" ]]; then
+ sh -c 'echo $$ > /tmp/controller.pid; exec ./pinot-admin.sh StartController -zkAddress localhost:$0 -clusterName PinotCluster -controllerPort $1 > /tmp/pinot-controller.log' ${localhost} "${controllerPort}"
Review comment:
Just start with default ports for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org