You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gengliangwang <gi...@git.apache.org> on 2018/10/15 13:55:37 UTC
[GitHub] spark pull request #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftser...
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/22727
[SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh: print clean usage and exit with code 1
## What changes were proposed in this pull request?
Currently if we run
```
sh start-thriftserver.sh -h
```
we get
```
...
Thrift server options:
2018-10-15 21:45:39 INFO HiveThriftServer2:54 - Starting SparkContext
2018-10-15 21:45:40 INFO SparkContext:54 - Running Spark version 2.3.2
2018-10-15 21:45:40 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-10-15 21:45:40 ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:367)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2493)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:48)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:79)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
2018-10-15 21:45:40 ERROR Utils:91 - Uncaught exception in thread main
```
After fix, the usage output is clean:
```
...
Thrift server options:
--hiveconf <property=value> Use value for given property
```
Also exit with code 1, to follow other scripts(this is the behavior of parsing option `-h` for other linux commands as well).
## How was this patch tested?
Manual test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gengliangwang/spark stsUsage
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22727.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22727
----
commit 3c84e3a4d182e0bf9553ba74b7398b169c3d65bd
Author: Gengliang Wang <ge...@...>
Date: 2018-10-15T13:27:25Z
imrove printing usage
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftser...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22727#discussion_r225175351
--- Diff: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala ---
@@ -71,6 +71,12 @@ object HiveThriftServer2 extends Logging {
}
def main(args: Array[String]) {
+ // If the arguments contains "-h" or "--help", print out the usage and exit.
+ if (args.contains("-h") || args.contains("--help")) {
+ HiveServer2.main(args)
--- End diff --
This is a bit ugly, but it is the only solution I can think of.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97394/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22727
**[Test build #97396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97396/testReport)** for PR 22727 at commit [`65da8a6`](https://github.com/apache/spark/commit/65da8a621192aac34ed22b937825a2a6cb2a8c8e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/22727
Merged to master
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22727
**[Test build #97394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97394/testReport)** for PR 22727 at commit [`3c84e3a`](https://github.com/apache/spark/commit/3c84e3a4d182e0bf9553ba74b7398b169c3d65bd).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftser...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22727
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22727
@srowen @dongjoon-hyun @jiangxb1987
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22727
**[Test build #97394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97394/testReport)** for PR 22727 at commit [`3c84e3a`](https://github.com/apache/spark/commit/3c84e3a4d182e0bf9553ba74b7398b169c3d65bd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3988/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftser...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22727#discussion_r225179121
--- Diff: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala ---
@@ -71,6 +71,12 @@ object HiveThriftServer2 extends Logging {
}
def main(args: Array[String]) {
+ // If the arguments contains "-h" or "--help", print out the usage and exit.
+ if (args.contains("-h") || args.contains("--help")) {
+ HiveServer2.main(args)
--- End diff --
That's reasonable. But just return in the line below. System.exit is unnecessarily abrupt.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftser...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/22727#discussion_r225417566
--- Diff: sbin/start-thriftserver.sh ---
@@ -39,6 +39,10 @@ function usage {
pattern+="\|Spark Command: "
pattern+="\|======="
pattern+="\|--help"
+ pattern+="\|Using Spark's default log4j profile:"
+ pattern+="\|^log4j:"
+ pattern+="\|Started daemon with process name"
+ pattern+="\|Registered signal handler for"
"${SPARK_HOME}"/bin/spark-submit --help 2>&1 | grep -v Usage 1>&2
--- End diff --
it;s kinda weird that `--help` will try to create the context?
`ERROR SparkContext:91 - Error initializing SparkContext.`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4019/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22727
**[Test build #97430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97430/testReport)** for PR 22727 at commit [`8b2500f`](https://github.com/apache/spark/commit/8b2500fa97ab258785160df25d3a0f6afdd74a10).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97430/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97396/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22727
**[Test build #97430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97430/testReport)** for PR 22727 at commit [`8b2500f`](https://github.com/apache/spark/commit/8b2500fa97ab258785160df25d3a0f6afdd74a10).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftser...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22727#discussion_r225392908
--- Diff: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala ---
@@ -71,6 +71,12 @@ object HiveThriftServer2 extends Logging {
}
def main(args: Array[String]) {
+ // If the arguments contains "-h" or "--help", print out the usage and exit.
+ if (args.contains("-h") || args.contains("--help")) {
+ HiveServer2.main(args)
--- End diff --
I see. I was following `HiveServer2` to call `System.exit`.
Now I change to `return`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22727
**[Test build #97396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97396/testReport)** for PR 22727 at commit [`65da8a6`](https://github.com/apache/spark/commit/65da8a621192aac34ed22b937825a2a6cb2a8c8e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftser...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22727#discussion_r225418151
--- Diff: sbin/start-thriftserver.sh ---
@@ -39,6 +39,10 @@ function usage {
pattern+="\|Spark Command: "
pattern+="\|======="
pattern+="\|--help"
+ pattern+="\|Using Spark's default log4j profile:"
+ pattern+="\|^log4j:"
+ pattern+="\|Started daemon with process name"
+ pattern+="\|Registered signal handler for"
"${SPARK_HOME}"/bin/spark-submit --help 2>&1 | grep -v Usage 1>&2
--- End diff --
Yes, the error message is gone after the code change.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3986/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22727: [SPARK-25735][CORE][MINOR]Improve start-thriftserver.sh:...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22727
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org