You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by denalex <gi...@git.apache.org> on 2017/02/08 21:50:12 UTC
[GitHub] incubator-hawq pull request #1109: HAWQ-1298 Refactored RPS management scrip...
GitHub user denalex opened a pull request:
https://github.com/apache/incubator-hawq/pull/1109
HAWQ-1298 Refactored RPS management scripts
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/denalex/incubator-hawq HAWQ-1298
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/1109.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1109
----
commit 920f490a3afb6aeaecbf6bbdb4ce345537953b84
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-01-31T22:31:06Z
HAWQ-1298. Updated RPS properties and scripts
commit 06ed4444ed7a11828b38003d5bba27c8015eec26
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-02T00:26:08Z
refactored scripts
commit b9ef79b15b82f7823a396f74e74c1f005c82e0b8
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-02T00:30:11Z
renamed register_hawq.sh
commit 1d279032226fe9bdcef345e2b497be5864b0ea95
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-02T21:55:39Z
added catalina.sh
commit 166287e7d7f36eec68b5a19dad07223f1ddfe7dc
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-02T22:38:43Z
added POLICY_MGR_URL property
commit 1ac816139d3ac6b1d4d638b0ca087d14cfd7dcaa
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-02T22:53:56Z
fixed long string
commit aece77883795087df9ca7fad71aaac6770c3a7af
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-03T00:21:15Z
substitution of POLICY_MGR_URL
commit dfd1ec6be850baaf6be78a238b3d890d32e8a714
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-03T00:28:40Z
renamed the script
commit 23e5bcc21dc5a5f4536850e4b7f75cb38bcfffcd
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-03T01:30:40Z
updated sed separation character
commit 69466044c3d461aceec13611926bc314eea28692
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-03T01:32:18Z
updated prop_file variable
commit 5dc98e5b9f138ca298907e019b80d0a539a1cdbd
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-03T20:54:23Z
defined log files in log4j.properties
commit 34e546838e072ed70e6158d9ba8ddd8d099bc2a9
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-03T22:11:16Z
added auditing
commit 68618296e02cdf136bdfcae76a2aa1f711e00ebf
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-03T22:22:05Z
fixed additivity
commit 3ccc6a64f564bc36acdf694a388358c0c58fffbb
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-03T22:49:00Z
added hadoop-common.jar to lib dir
commit 585797f718afdcfafbf8b10cf3db89904a5d1ace
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-03T22:54:37Z
fixed typo
commit 93bdbc55e9862d8c990a4e8917fc840e1c810880
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-03T23:14:24Z
inlude hadoop-common.jar
commit 195e8d291b8359406bdb8275651b6ba4f17eba9e
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-07T21:39:31Z
added catalina.props, not shipping JAR files in WAR
commit cbc53c4d8534e1c481a08afd7a84ff6518eb50b3
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-08T19:14:44Z
removed db audit
commit 2d8267a908707d6af6f854f12f85afb507cba8b7
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-08T19:38:57Z
removed filemode from RPM
commit b2f83c8d5b6447166cffab512271aae8fed60366
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-08T19:51:30Z
added filemode to RPM
commit e799003c8fceaf34ae4df41e5f0655d63be148c9
Author: Alexander Denissov <ad...@pivotal.io>
Date: 2017-02-08T21:46:43Z
restored gpadmin user for RPM
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1109: HAWQ-1298 Refactored RPS management scripts
Posted by zhangh43 <gi...@git.apache.org>.
Github user zhangh43 commented on the issue:
https://github.com/apache/incubator-hawq/pull/1109
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1109: HAWQ-1298 Refactored RPS management scrip...
Posted by zhangh43 <gi...@git.apache.org>.
Github user zhangh43 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1109#discussion_r100270470
--- Diff: ranger-plugin/scripts/rps.sh ---
@@ -20,41 +20,71 @@
#
if [ $# -le 0 ]; then
- echo "Usage: rps (start|stop|init) [<catalina-args...>]"
+ echo "Usage: rps.sh (start|stop)"
exit 1
fi
-actionCmd=$1
+action=$1
shift
-CWDIR=$( cd $( dirname ${BASH_SOURCE[0]} ) && pwd )
-source $CWDIR/rps_env.sh
-
-setup_rps() {
- echo "Initializing Hawq Ranger Plugin Service..."
- cp $CATALINA_HOME/conf.template/* $CATALINA_BASE/conf
- cp $CATALINA_BASE/conf/tomcat-server.xml $CATALINA_BASE/conf/server.xml
- pushd $CATALINA_BASE/webapps >/dev/null
- unzip -d rps rps.war >/dev/null
- find . -name ranger-hawq-security.xml | xargs sed -i \
- "s/localhost:6080/$RANGER_ADMIN_HOST:$RANGER_ADMIN_PORT/g"
- popd >/dev/null
- echo "Hawq Ranger Plugin Service installed on http://$RPS_HOST:$RPS_PORT/rps"
- echo "Please use 'rps.sh start' to start the service"
+CWDIR=$( cd $( dirname ${BASH_SOURCE[0]} ) && pwd -P)
+BASEDIR=$( dirname ${CWDIR} )
+# read properties from the file
+source ${BASEDIR}/etc/rps.properties
+
+export CATALINA_HOME=/usr/lib/bigtop-tomcat
+export CATALINA_BASE=${BASEDIR}/plugin-service
+export CATALINA_PID=${CATALINA_BASE}/work/rps.pid
+
+# options used to start the RPS process
+export CATALINA_OPTS="-server -Xms512m -Xmx512m -XX:MaxPermSize=128m
+ -Dproc_rps -Dversion=${RPS_VERSION}
+ -Dranger.hawq.instance=${RANGER_HAWQ_INSTANCE}
+ -Drps.http.port=${RPS_HTTP_PORT} -Drps.https.port=${RPS_HTTPS_PORT}
+ -Dpolicy.manager.url=${POLICY_MGR_URL}"
+
+# options used to stop the RPS process
+export JAVA_OPTS="-Drps.shutdown.port=${RPS_SHUTDOWN_PORT}"
+
+RPS_URL="http://localhost:${RPS_HTTP_PORT}/rps"
+RPS_LOG="${CATALINA_BASE}/logs/catalina.out"
+
+function fail() {
+ echo "FATAL: Failed to ${1} HAWQ Ranger Plugin Service. Check ${RPS_LOG} for details."
+ exit 2
}
-case $actionCmd in
- (init)
- setup_rps
- ;;
+function tomcat_command() {
+ ${CWDIR}/catalina.sh ${1} ${2}
+ if [ $? -ne 0 ]; then
+ fail ${1}
+ fi
+}
+
+function wait_until_server_started() {
+ echo -n "Waiting for Hawq Ranger Plugin Service to start ."
+ local retries="20"
+ local n=0
+ until $(curl -s --output /dev/null --fail ${RPS_URL}/version); do
--- End diff --
Got it. When server is down, curl --fail will return 22. When server is up, it return 0 successfully.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1109: HAWQ-1298 Refactored RPS management scrip...
Posted by zhangh43 <gi...@git.apache.org>.
Github user zhangh43 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1109#discussion_r100223135
--- Diff: ranger-plugin/integration/service/src/test/java/org/apache/hawq/ranger/integration/service/tests/common/ServiceTestBase.java ---
@@ -67,7 +67,8 @@
private static final String RANGER_URL = String.format("http://%s:%s/service/public/v2/api", RANGER_HOST, RANGER_PORT);
private static final String RANGER_POLICY_URL = RANGER_URL + "/policy";
- private static final int POLICY_REFRESH_INTERVAL = 6000;
+ private static final String POLICY_WAIT_INTERVAL_PROP_NAME = "policy.wait.interval.ms";
--- End diff --
could you explain what is the difference between ranger.plugin.hawq.policy.pollIntervalMs and policy.wait.interval.ms? And in which xml file to config policy.wait.interval.ms?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1109: HAWQ-1298 Refactored RPS management scrip...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-hawq/pull/1109
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1109: HAWQ-1298 Refactored RPS management scrip...
Posted by zhangh43 <gi...@git.apache.org>.
Github user zhangh43 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1109#discussion_r100270687
--- Diff: ranger-plugin/integration/service/src/test/java/org/apache/hawq/ranger/integration/service/tests/common/ServiceTestBase.java ---
@@ -67,7 +67,8 @@
private static final String RANGER_URL = String.format("http://%s:%s/service/public/v2/api", RANGER_HOST, RANGER_PORT);
private static final String RANGER_POLICY_URL = RANGER_URL + "/policy";
- private static final int POLICY_REFRESH_INTERVAL = 6000;
+ private static final String POLICY_WAIT_INTERVAL_PROP_NAME = "policy.wait.interval.ms";
--- End diff --
Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1109: HAWQ-1298 Refactored RPS management scrip...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1109#discussion_r100231398
--- Diff: ranger-plugin/integration/service/src/test/java/org/apache/hawq/ranger/integration/service/tests/common/ServiceTestBase.java ---
@@ -67,7 +67,8 @@
private static final String RANGER_URL = String.format("http://%s:%s/service/public/v2/api", RANGER_HOST, RANGER_PORT);
private static final String RANGER_POLICY_URL = RANGER_URL + "/policy";
- private static final int POLICY_REFRESH_INTERVAL = 6000;
+ private static final String POLICY_WAIT_INTERVAL_PROP_NAME = "policy.wait.interval.ms";
--- End diff --
ranger.plugin.hawq.policy.pollIntervalMs is a real property used by the service and determines the frequency of policy refreshes performed by Ranger Plugin.
policy.wait.interval.ms is the property used only by integration test suite to wait between updating policy in Ranger and trying to evaluate the request. In our tests we currently set wait.interval = 3 * pollInterval, even that sometimes causes random test failures when the system is under heavy load. We can optimize this further, if needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1109: HAWQ-1298 Refactored RPS management scrip...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1109#discussion_r100231177
--- Diff: ranger-plugin/scripts/rps.sh ---
@@ -20,41 +20,71 @@
#
if [ $# -le 0 ]; then
- echo "Usage: rps (start|stop|init) [<catalina-args...>]"
+ echo "Usage: rps.sh (start|stop)"
exit 1
fi
-actionCmd=$1
+action=$1
shift
-CWDIR=$( cd $( dirname ${BASH_SOURCE[0]} ) && pwd )
-source $CWDIR/rps_env.sh
-
-setup_rps() {
- echo "Initializing Hawq Ranger Plugin Service..."
- cp $CATALINA_HOME/conf.template/* $CATALINA_BASE/conf
- cp $CATALINA_BASE/conf/tomcat-server.xml $CATALINA_BASE/conf/server.xml
- pushd $CATALINA_BASE/webapps >/dev/null
- unzip -d rps rps.war >/dev/null
- find . -name ranger-hawq-security.xml | xargs sed -i \
- "s/localhost:6080/$RANGER_ADMIN_HOST:$RANGER_ADMIN_PORT/g"
- popd >/dev/null
- echo "Hawq Ranger Plugin Service installed on http://$RPS_HOST:$RPS_PORT/rps"
- echo "Please use 'rps.sh start' to start the service"
+CWDIR=$( cd $( dirname ${BASH_SOURCE[0]} ) && pwd -P)
+BASEDIR=$( dirname ${CWDIR} )
+# read properties from the file
+source ${BASEDIR}/etc/rps.properties
+
+export CATALINA_HOME=/usr/lib/bigtop-tomcat
+export CATALINA_BASE=${BASEDIR}/plugin-service
+export CATALINA_PID=${CATALINA_BASE}/work/rps.pid
+
+# options used to start the RPS process
+export CATALINA_OPTS="-server -Xms512m -Xmx512m -XX:MaxPermSize=128m
+ -Dproc_rps -Dversion=${RPS_VERSION}
+ -Dranger.hawq.instance=${RANGER_HAWQ_INSTANCE}
+ -Drps.http.port=${RPS_HTTP_PORT} -Drps.https.port=${RPS_HTTPS_PORT}
+ -Dpolicy.manager.url=${POLICY_MGR_URL}"
+
+# options used to stop the RPS process
+export JAVA_OPTS="-Drps.shutdown.port=${RPS_SHUTDOWN_PORT}"
+
+RPS_URL="http://localhost:${RPS_HTTP_PORT}/rps"
+RPS_LOG="${CATALINA_BASE}/logs/catalina.out"
+
+function fail() {
+ echo "FATAL: Failed to ${1} HAWQ Ranger Plugin Service. Check ${RPS_LOG} for details."
+ exit 2
}
-case $actionCmd in
- (init)
- setup_rps
- ;;
+function tomcat_command() {
+ ${CWDIR}/catalina.sh ${1} ${2}
+ if [ $? -ne 0 ]; then
+ fail ${1}
+ fi
+}
+
+function wait_until_server_started() {
+ echo -n "Waiting for Hawq Ranger Plugin Service to start ."
+ local retries="20"
+ local n=0
+ until $(curl -s --output /dev/null --fail ${RPS_URL}/version); do
--- End diff --
It works, I think when you tested you had RPS shut down. Curl returns error 7 if it can't connect to host. When RPS is up, the return is 0. See the trace below:
bash-4.1# curl -s --output /dev/null --fail http://localhost:8432/rps/version
bash-4.1# echo $?
7
bash-4.1# curl --fail http://localhost:8432/rps/version
curl: (7) couldn't connect to host
bash-4.1# ps aux | grep rps
root 2340 0.0 0.0 6500 464 ? R+ 04:58 0:00 grep rps
bash-4.1# ./rps.sh start
Using CATALINA_BASE: /usr/local/hawq_2_1_0_0/ranger/plugin-service
Using CATALINA_HOME: /usr/lib/bigtop-tomcat
Using CATALINA_TMPDIR: /usr/local/hawq_2_1_0_0/ranger/plugin-service/temp
Using JRE_HOME: /usr/lib/jvm/java-1.7.0-openjdk.x86_64
Using CLASSPATH: /usr/local/hawq_2_1_0_0/ranger/etc:/usr/lib/bigtop-tomcat/bin/bootstrap.jar
Using CATALINA_PID: /usr/local/hawq_2_1_0_0/ranger/plugin-service/work/rps.pid
Waiting for Hawq Ranger Plugin Service to start ..
Hawq Ranger Plugin Service is available at http://localhost:8432/rps
bash-4.1# curl -s --output /dev/null --fail http://localhost:8432/rps/version
bash-4.1# echo $?
0
bash-4.1#
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1109: HAWQ-1298 Refactored RPS management scrip...
Posted by zhangh43 <gi...@git.apache.org>.
Github user zhangh43 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1109#discussion_r100222596
--- Diff: ranger-plugin/scripts/rps.sh ---
@@ -20,41 +20,71 @@
#
if [ $# -le 0 ]; then
- echo "Usage: rps (start|stop|init) [<catalina-args...>]"
+ echo "Usage: rps.sh (start|stop)"
exit 1
fi
-actionCmd=$1
+action=$1
shift
-CWDIR=$( cd $( dirname ${BASH_SOURCE[0]} ) && pwd )
-source $CWDIR/rps_env.sh
-
-setup_rps() {
- echo "Initializing Hawq Ranger Plugin Service..."
- cp $CATALINA_HOME/conf.template/* $CATALINA_BASE/conf
- cp $CATALINA_BASE/conf/tomcat-server.xml $CATALINA_BASE/conf/server.xml
- pushd $CATALINA_BASE/webapps >/dev/null
- unzip -d rps rps.war >/dev/null
- find . -name ranger-hawq-security.xml | xargs sed -i \
- "s/localhost:6080/$RANGER_ADMIN_HOST:$RANGER_ADMIN_PORT/g"
- popd >/dev/null
- echo "Hawq Ranger Plugin Service installed on http://$RPS_HOST:$RPS_PORT/rps"
- echo "Please use 'rps.sh start' to start the service"
+CWDIR=$( cd $( dirname ${BASH_SOURCE[0]} ) && pwd -P)
+BASEDIR=$( dirname ${CWDIR} )
+# read properties from the file
+source ${BASEDIR}/etc/rps.properties
+
+export CATALINA_HOME=/usr/lib/bigtop-tomcat
+export CATALINA_BASE=${BASEDIR}/plugin-service
+export CATALINA_PID=${CATALINA_BASE}/work/rps.pid
+
+# options used to start the RPS process
+export CATALINA_OPTS="-server -Xms512m -Xmx512m -XX:MaxPermSize=128m
+ -Dproc_rps -Dversion=${RPS_VERSION}
+ -Dranger.hawq.instance=${RANGER_HAWQ_INSTANCE}
+ -Drps.http.port=${RPS_HTTP_PORT} -Drps.https.port=${RPS_HTTPS_PORT}
+ -Dpolicy.manager.url=${POLICY_MGR_URL}"
+
+# options used to stop the RPS process
+export JAVA_OPTS="-Drps.shutdown.port=${RPS_SHUTDOWN_PORT}"
+
+RPS_URL="http://localhost:${RPS_HTTP_PORT}/rps"
+RPS_LOG="${CATALINA_BASE}/logs/catalina.out"
+
+function fail() {
+ echo "FATAL: Failed to ${1} HAWQ Ranger Plugin Service. Check ${RPS_LOG} for details."
+ exit 2
}
-case $actionCmd in
- (init)
- setup_rps
- ;;
+function tomcat_command() {
+ ${CWDIR}/catalina.sh ${1} ${2}
+ if [ $? -ne 0 ]; then
+ fail ${1}
+ fi
+}
+
+function wait_until_server_started() {
+ echo -n "Waiting for Hawq Ranger Plugin Service to start ."
+ local retries="20"
+ local n=0
+ until $(curl -s --output /dev/null --fail ${RPS_URL}/version); do
--- End diff --
The --fail option should be removed, its return code is 22.
Since the logic here is to repeat waiting if rps is not ready. When RPS is ready, return code should be 0, and until statement returns.
curl -s --output /dev/null --fail ${Right_RPS_URL}/version
echo $? => 22
curl -s --output /dev/null ${Right_RPS_URL}/version
echo $? => 0
curl -s --output /dev/null --fail ${Wrong_RPS_URL}/version
echo $? => 7
curl -s --output /dev/null ${Right_RPS_URL}/version
echo $? => 7
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---