You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JoshRosen <gi...@git.apache.org> on 2016/01/24 01:16:06 UTC
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/10884
[SPARK-12971][test-hadoop2.3] Try to fix Hive tests which fail in Hadoop-2.3 SBT build
ErrorPositionSuite and one of the HiveComparisonTest tests have been consistently failing on the Hadoop 2.3 SBT build (but on no other builds). I believe that this is due to test isolation issues (e.g. tests sharing state via the sets of temporary tables that are registered to TestHive).
This patch attempts to improve the isolation of these tests in order to address this issue.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark fix-failing-hadoop-2.3-hive-tests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10884.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10884
----
commit 2f36e05eb5ff041ca97f30207339dd9035a0bcc9
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-01-23T22:57:23Z
Try to fix Hive tests which fail in Hadoop-2.3 SBT
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.2] Fix Hive tests w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174253899
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49947/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.4] Fix Hive tests w...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174244744
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.4] Fix Hive tests w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174245273
**[Test build #49946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49946/consoleFull)** for PR 10884 at commit [`2f36e05`](https://github.com/apache/spark/commit/2f36e05eb5ff041ca97f30207339dd9035a0bcc9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.2] Fix Hive tests w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174252874
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.2] Fix Hive tests w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174252877
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49946/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174243492
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.2] Fix Hive tests w...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174245424
jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.2] Fix Hive tests w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174246212
**[Test build #49947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49947/consoleFull)** for PR 10884 at commit [`2f36e05`](https://github.com/apache/spark/commit/2f36e05eb5ff041ca97f30207339dd9035a0bcc9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.2] Fix Hive tests w...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174253898
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174238407
**[Test build #49942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49942/consoleFull)** for PR 10884 at commit [`2f36e05`](https://github.com/apache/spark/commit/2f36e05eb5ff041ca97f30207339dd9035a0bcc9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.4] Fix Hive tests w...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174244757
(Retesting to see whether this also addresses a SBT Hadoop 2.4 failure, which I suspect has the same root cause)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.2] Fix Hive tests w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174253850
**[Test build #49947 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49947/consoleFull)** for PR 10884 at commit [`2f36e05`](https://github.com/apache/spark/commit/2f36e05eb5ff041ca97f30207339dd9035a0bcc9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfterEach `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/10884#discussion_r50627893
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/ErrorPositionSuite.scala ---
@@ -19,20 +19,34 @@ package org.apache.spark.sql.hive
import scala.util.Try
-import org.scalatest.BeforeAndAfter
+import org.scalatest.BeforeAndAfterEach
import org.apache.spark.sql.{AnalysisException, QueryTest}
import org.apache.spark.sql.catalyst.parser.ParseDriver
import org.apache.spark.sql.catalyst.util.quietly
import org.apache.spark.sql.hive.test.TestHiveSingleton
-class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfter {
+class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfterEach {
import hiveContext.implicits._
- before {
+ override protected def beforeEach(): Unit = {
+ super.beforeEach()
+ if (sqlContext.tableNames().contains("src")) {
+ sqlContext.dropTempTable("src")
+ }
+ Seq((1, "")).toDF("key", "value").registerTempTable("src")
--- End diff --
Nevermind. The ```src``` attributes are also resolved in some tests...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/10884#discussion_r50627843
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/ErrorPositionSuite.scala ---
@@ -19,20 +19,34 @@ package org.apache.spark.sql.hive
import scala.util.Try
-import org.scalatest.BeforeAndAfter
+import org.scalatest.BeforeAndAfterEach
import org.apache.spark.sql.{AnalysisException, QueryTest}
import org.apache.spark.sql.catalyst.parser.ParseDriver
import org.apache.spark.sql.catalyst.util.quietly
import org.apache.spark.sql.hive.test.TestHiveSingleton
-class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfter {
+class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfterEach {
import hiveContext.implicits._
- before {
+ override protected def beforeEach(): Unit = {
+ super.beforeEach()
+ if (sqlContext.tableNames().contains("src")) {
+ sqlContext.dropTempTable("src")
+ }
+ Seq((1, "")).toDF("key", "value").registerTempTable("src")
--- End diff --
We are testing query positionering errors here. The query is probably deemed invalid before we check that the ```src``` table exists.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10884#discussion_r50627794
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala ---
@@ -150,7 +150,11 @@ abstract class HiveComparisonTest
""".stripMargin
})
- super.afterAll()
+ try {
+ TestHive.reset()
--- End diff --
It doesn't hurt to throw in an extra `reset()` at the end of suite teardown here and might happen to fix some other isolation issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174243493
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49942/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10884#discussion_r50627787
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/ErrorPositionSuite.scala ---
@@ -19,20 +19,34 @@ package org.apache.spark.sql.hive
import scala.util.Try
-import org.scalatest.BeforeAndAfter
+import org.scalatest.BeforeAndAfterEach
import org.apache.spark.sql.{AnalysisException, QueryTest}
import org.apache.spark.sql.catalyst.parser.ParseDriver
import org.apache.spark.sql.catalyst.util.quietly
import org.apache.spark.sql.hive.test.TestHiveSingleton
-class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfter {
+class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfterEach {
import hiveContext.implicits._
- before {
+ override protected def beforeEach(): Unit = {
+ super.beforeEach()
+ if (sqlContext.tableNames().contains("src")) {
+ sqlContext.dropTempTable("src")
+ }
+ Seq((1, "")).toDF("key", "value").registerTempTable("src")
--- End diff --
When this suite was first added, I believe that `src` referred to a temporary table loaded in TestHive, which is why I used its schema here.
Interestingly, all of the tests in this suite pass even if `src` doesn't exist, which suggests to me that the tests aren't strong enough to detect certain classes of bugs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971] Fix Hive tests which fail in Had...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10884
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10884#discussion_r50627984
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/ErrorPositionSuite.scala ---
@@ -19,20 +19,34 @@ package org.apache.spark.sql.hive
import scala.util.Try
-import org.scalatest.BeforeAndAfter
+import org.scalatest.BeforeAndAfterEach
import org.apache.spark.sql.{AnalysisException, QueryTest}
import org.apache.spark.sql.catalyst.parser.ParseDriver
import org.apache.spark.sql.catalyst.util.quietly
import org.apache.spark.sql.hive.test.TestHiveSingleton
-class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfter {
+class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfterEach {
import hiveContext.implicits._
- before {
+ override protected def beforeEach(): Unit = {
+ super.beforeEach()
+ if (sqlContext.tableNames().contains("src")) {
+ sqlContext.dropTempTable("src")
+ }
+ Seq((1, "")).toDF("key", "value").registerTempTable("src")
--- End diff --
As an experiment, I just tried adding a new test which drops the `src` table, then runs
```
positionTest("unresolved attribute new test",
"""SELECT x,
|key FROM src
""".stripMargin, "key")
```
and it looks like this actually does fail with
```
[info] - unresolved attribute 4 *** FAILED *** (47 milliseconds)
[info] 1 did not equal 2 wrong line (ErrorPositionSuite.scala:153)
```
I guess that the attribute resolution order doesn't relate to the order in which the attributes appear in the query: I think `x` is always resolved before `key`, which is why this test always passes irrespective of whether `key` exists.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174243409
**[Test build #49942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49942/consoleFull)** for PR 10884 at commit [`2f36e05`](https://github.com/apache/spark/commit/2f36e05eb5ff041ca97f30207339dd9035a0bcc9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfterEach `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10884#discussion_r50627993
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/ErrorPositionSuite.scala ---
@@ -19,20 +19,34 @@ package org.apache.spark.sql.hive
import scala.util.Try
-import org.scalatest.BeforeAndAfter
+import org.scalatest.BeforeAndAfterEach
import org.apache.spark.sql.{AnalysisException, QueryTest}
import org.apache.spark.sql.catalyst.parser.ParseDriver
import org.apache.spark.sql.catalyst.util.quietly
import org.apache.spark.sql.hive.test.TestHiveSingleton
-class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfter {
+class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfterEach {
import hiveContext.implicits._
- before {
+ override protected def beforeEach(): Unit = {
+ super.beforeEach()
+ if (sqlContext.tableNames().contains("src")) {
+ sqlContext.dropTempTable("src")
+ }
+ Seq((1, "")).toDF("key", "value").registerTempTable("src")
--- End diff --
However, this limitation in the existing test is out-of-scope for the purposes of this patch, which is just to fix Jenkins so my test failure notifications stop blowing up with notifications about this suite (I get a chat message for every test failure).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.2] Fix Hive tests w...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10884#issuecomment-174252486
**[Test build #49946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49946/consoleFull)** for PR 10884 at commit [`2f36e05`](https://github.com/apache/spark/commit/2f36e05eb5ff041ca97f30207339dd9035a0bcc9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfterEach `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12971][test-hadoop2.3] Try to fix Hive ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10884#discussion_r50627933
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/ErrorPositionSuite.scala ---
@@ -19,20 +19,34 @@ package org.apache.spark.sql.hive
import scala.util.Try
-import org.scalatest.BeforeAndAfter
+import org.scalatest.BeforeAndAfterEach
import org.apache.spark.sql.{AnalysisException, QueryTest}
import org.apache.spark.sql.catalyst.parser.ParseDriver
import org.apache.spark.sql.catalyst.util.quietly
import org.apache.spark.sql.hive.test.TestHiveSingleton
-class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfter {
+class ErrorPositionSuite extends QueryTest with TestHiveSingleton with BeforeAndAfterEach {
import hiveContext.implicits._
- before {
+ override protected def beforeEach(): Unit = {
+ super.beforeEach()
+ if (sqlContext.tableNames().contains("src")) {
+ sqlContext.dropTempTable("src")
+ }
+ Seq((1, "")).toDF("key", "value").registerTempTable("src")
--- End diff --
I was thinking about the following tests:
```scala
positionTest("unresolved attribute 1",
"SELECT x FROM src", "x")
```
and
```scala
positionTest("unresolved attribute 3",
"SELECT key, x FROM src", "x")
```
and
```scala
positionTest("unresolved attribute 4",
"""SELECT key,
|x FROM src
""".stripMargin, "x")
```
In the second and third tests, it looks like the assert is trying to check that "x" is used when positioning the error message and not "key". Therefore, I think it matters that the table has a column named "key" in order for this test to be meaningful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org