You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ckadner <gi...@git.apache.org> on 2015/10/29 02:45:15 UTC
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
GitHub user ckadner opened a pull request:
https://github.com/apache/spark/pull/9345
[SPARK-10890][SQL] don't log expected JdbcSQLException during incompatible Insert test
The `org.h2.jdbc.JdbcSQLException` is logged by `org.apache.spark.executor.Executor` and again by `org.apache.spark.scheduler.TaskSetManager`, so we would need to influence their log behavior / log levels.
Absent any properties or API to do that, the easiest way to suppress the logging of the expected exception is to temporarily increase the log threshold to `FATAL` while the test case is executed and restore the original log level right after. This could be done globally `org.apache.log4j.RootLogger` or just for `org.apache.spark.executor.Executor` and `org.apache.spark.scheduler.TaskSetManager`
I realize this is a _"hack"_ with the side effect of suppressing any other log message of less than FATAL severity during the test case execution. Discussion invited :-)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ckadner/spark SPARK-10890
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9345.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9345
----
commit 470fd09b1de528225e1673621a14b332ae1e6e0b
Author: Christian Kadner <ck...@us.ibm.com>
Date: 2015-10-29T01:23:20Z
[SPARK-10890][SQL] don't log expected JdbcSQLException during incompatible Insert test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152423150
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by rick-ibm <gi...@git.apache.org>.
Github user rick-ibm commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152355615
Hi Christian,
The patch looks good to me. I would recommend putting the restoration of the logging state inside a finally block. I don't know if that will change the behavior of this test, but it seems like good future-proofing in case someone else wants to copy your approach later on.
Thanks,
-Rick
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152421377
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152357861
@**Rick**, I was running the `JDBCWriteSuite` in IntelliJ and each test case got setup with log `Level.DEBUG`, even if an unexpected `Exception` prevented resetting the log level to it's original.
But your's is a very prudent suggestion and I will push an update shortly -- **Thanks**
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-174336317
Ping @ckadner, would you mind closing this PR per Yin's comment? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152362181
+1 @JoshRosen -- Good suggestion! I pushed a new version.
Could you kick of a test build :mans_shoe:
Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152364166
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152373974
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44650/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152450742
**[Test build #44663 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44663/consoleFull)** for PR 9345 at commit [`a2e236f`](https://github.com/apache/spark/commit/a2e236f5d45c6e93608d128bf9556dfd227919d6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-171390193
@ckadner Thank you for the PR. Since it does not really affect tests and the solution at here is kind of hacky, how about we close this PR for now and revisit it later if we have a better approach? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152049348
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152450893
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152400158
Hi @JoshRosen, the failing unit test `"Single command with -e"` in `CliSuite` should not be impacted by my changes. On my machine I get the timeouts with or without my changes. Is this a known problem?
```
[info] *** 1 TEST FAILED ***
[error] Failed: Total 19, Failed 1, Errors 0, Passed 18, Ignored 1
[error] Failed tests:
[error] org.apache.spark.sql.hive.thriftserver.CliSuite
[error] (hive-thriftserver/test:test) sbt.TestsFailedException: Tests unsuccessful
...
[info] CliSuite:
[info] - Simple commands (1 minute, 35 seconds)
[info] - Single command with -e *** FAILED *** (2 minutes)
[info] =======================
[info] CliSuite failure output
[info] =======================
[info] Spark SQL CLI command line: ../../bin/spark-sql --master local --hiveconf javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-467fd8f3-b708-4fbc-8663-1486857bdb0d;create=true --hiveconf hive.metastore.warehouse.dir=/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-a45eff3e-fc0f-461f-b658-471176d83655 --hiveconf hive.exec.scratchdir=/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-869dc65d-560d-4df5-b6d4-7cd673974013 -e SHOW DATABASES;
[info] Exception: java.util.concurrent.TimeoutException: Futures timed out after [2 minutes]
[info] Executed query 0 "",
[info] But failed to capture expected output "OK" within 2 minutes.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152373973
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152421379
It's a known flaky test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152450896
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44663/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152373946
**[Test build #44650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44650/consoleFull)** for PR 9345 at commit [`a2e236f`](https://github.com/apache/spark/commit/a2e236f5d45c6e93608d128bf9556dfd227919d6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by ckadner <gi...@git.apache.org>.
Github user ckadner closed the pull request at:
https://github.com/apache/spark/pull/9345
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152423996
**[Test build #44663 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44663/consoleFull)** for PR 9345 at commit [`a2e236f`](https://github.com/apache/spark/commit/a2e236f5d45c6e93608d128bf9556dfd227919d6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9345#discussion_r43457449
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCWriteSuite.scala ---
@@ -75,6 +76,17 @@ class JDBCWriteSuite extends SharedSQLContext with BeforeAndAfter {
conn1.close()
}
+ def runWithoutErrorLogging(function: () => Unit) {
--- End diff --
I'd use `=> Unit` so that you can use the nicer block syntax when calling this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152363574
Jenkins, this is ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152364190
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152423194
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10890][SQL] don't log expected JdbcSQLE...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9345#issuecomment-152367762
**[Test build #44650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44650/consoleFull)** for PR 9345 at commit [`a2e236f`](https://github.com/apache/spark/commit/a2e236f5d45c6e93608d128bf9556dfd227919d6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org