You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/19 07:06:29 UTC
[GitHub] [spark] sarutak opened a new pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
sarutak opened a new pull request #28578:
URL: https://github.com/apache/spark/pull/28578
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR mainly adds two things.
1. real headless browser support for UI test
2. A test suite using headless Chrome as one instance of those browsers.
Also, for environment where Chrome or Chrome driver is not installed, `ChromeUITest` tag is added to filter out the test suite.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
In the current master, there are two problems for UI test.
1. Lots of tests especially JavaScript related ones are done manually.
Appearance is better to be confirmed by our eyes but logic should be tested by test cases ideally.
2. Compared to the real web browsers, HtmlUnit doesn't seem to support JavaScriopt enough.
I added a JavaScript related test before for SPARK-31534 using HtmlUnit which is simple library based headless browser for test.
The test I added works somehow but some JavaScript related error is shown in unit-tests.log.
```
======= EXCEPTION START ========
Exception class=[net.sourceforge.htmlunit.corejs.javascript.JavaScriptException]
com.gargoylesoftware.htmlunit.ScriptException: Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". (http://192.168.1.209:60724/static/jquery-3.4.1.min.js#2)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:904)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:835)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:807)
at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216)
at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptFunctionJob.runJavaScript(JavaScriptFunctionJob.java:52)
at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptExecutionJob.run(JavaScriptExecutionJob.java:102)
at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptJobManagerImpl.runSingleJob(JavaScriptJobManagerImpl.java:426)
at com.gargoylesoftware.htmlunit.javascript.background.DefaultJavaScriptExecutor.run(DefaultJavaScriptExecutor.java:157)
at java.lang.Thread.run(Thread.java:748)
Caused by: net.sourceforge.htmlunit.corejs.javascript.JavaScriptException: Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". (http://192.168.1.209:60724/static/jquery-3.4.1.min.js#2)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1009)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:413)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:252)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$4.doRun(JavaScriptEngine.java:828)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:889)
... 10 more
JavaScriptException value = Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)".
== CALLING JAVASCRIPT ==
function () {
throw e;
}
======= EXCEPTION END ========
```
I tried to upgrade HtmlUnit to 2.40.0 but what is worse, the test become not working even though it works on real browsers like Chrome, Safari and Firefox without error.
```
[info] UISeleniumSuite:
[info] - SPARK-31534: text for tooltip should be escaped *** FAILED *** (17 seconds, 745 milliseconds)
[info] The code passed to eventually never returned normally. Attempted 2 times over 12.910785232 seconds. Last failure message: com.gargoylesoftware.htmlunit.ScriptException: ReferenceError: Assignment to undefined "regeneratorRuntime" in strict mode (http://192.168.1.209:62132/static/vis-timeline-graph2d.min.js#52(Function)#1)
```
To resolve those problems, it's better to support headless browser for UI test.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
I tested with following patterns.
1. sbt
`build/sbt -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite"`
2. sbt / filter test by tag
`build/sbt -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite -l org.apache.spark.tags.ChromeUITest"`
3. Maven
`build/mvn -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite test`
4. Maven / filter test by tag
`build/mvn -Dtest.exclude.tags="org.apache.spark.tags.ChromeUITest" -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite test`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633154382
I'll try it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-632688904
Merged to master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-630630582
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-631154654
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-631208800
**[Test build #122863 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122863/testReport)** for PR 28578 at commit [`6ccbc39`](https://github.com/apache/spark/commit/6ccbc391067995205127425ef59c4a9f1fe40520).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633157408
Thank you so much.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-631154272
**[Test build #122863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122863/testReport)** for PR 28578 at commit [`6ccbc39`](https://github.com/apache/spark/commit/6ccbc391067995205127425ef59c4a9f1fe40520).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-631209414
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633157536
Thank you. I saw the commit.
- https://github.com/apache/spark/commit/8441e936fc8ba9d4fc84385d1777c9e95a5e7636
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633153398
Hi, All.
This seems to break `Hadoop 3.2/Hive 2.3` profile Jenkins Maven jobs in master branch.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3/472/
...
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3/480/
```
ChromeUISeleniumSuite:
org.apache.spark.ui.ChromeUISeleniumSuite *** ABORTED ***
java.lang.IllegalStateException: The driver executable does not exist: /home/jenkins/workspace/spark-master-test-maven-hadoop-3.2-hive-2.3@2/core/null
at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
at org.openqa.selenium.remote.service.DriverService.checkExecutable(DriverService.java:121)
at org.openqa.selenium.remote.service.DriverService.findExecutable(DriverService.java:116)
at org.openqa.selenium.chrome.ChromeDriverService.access$000(ChromeDriverService.java:32)
at org.openqa.selenium.chrome.ChromeDriverService$Builder.findDefaultExecutable(ChromeDriverService.java:137)
at org.openqa.selenium.remote.service.DriverService$Builder.build(DriverService.java:296)
at org.openqa.selenium.chrome.ChromeDriverService.createDefaultService(ChromeDriverService.java:88)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:148)
at org.apache.spark.ui.ChromeUISeleniumSuite.beforeAll(ChromeUISeleniumSuite.scala:37)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
```
`Hadoop 3.2 / Hive 2.3 / JDK11` Jenkins Maven job is also failing together consistently.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3-jdk-11/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #28578:
URL: https://github.com/apache/spark/pull/28578#discussion_r427076227
##########
File path: dev/run-tests.py
##########
@@ -606,6 +609,8 @@ def main():
print("[info] Found the following changed modules:",
", ".join(x.name for x in changed_modules))
+ excluded_tags.extend(always_excluded_tags)
Review comment:
For environment where Chrome and Chrome driver is not installed, including CI environment, tests tagged as `ChromeUITest` is filtered out when tests are run through `run-tests.py`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-630731666
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #28578:
URL: https://github.com/apache/spark/pull/28578
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-631154272
**[Test build #122863 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122863/testReport)** for PR 28578 at commit [`6ccbc39`](https://github.com/apache/spark/commit/6ccbc391067995205127425ef59c4a9f1fe40520).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-630629927
**[Test build #122835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122835/testReport)** for PR 28578 at commit [`e701dc9`](https://github.com/apache/spark/commit/e701dc99048e63a4a8472d16eb70b2a41223ca45).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633153398
Hi, All.
This seems to break `Hadoop 3.2/Hive 2.3` profile Jenkins Maven jobs in master branch.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3/472/
...
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3/480/
```
ChromeUISeleniumSuite:
org.apache.spark.ui.ChromeUISeleniumSuite *** ABORTED ***
java.lang.IllegalStateException: The driver executable does not exist: /home/jenkins/workspace/spark-master-test-maven-hadoop-3.2-hive-2.3@2/core/null
at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
at org.openqa.selenium.remote.service.DriverService.checkExecutable(DriverService.java:121)
at org.openqa.selenium.remote.service.DriverService.findExecutable(DriverService.java:116)
at org.openqa.selenium.chrome.ChromeDriverService.access$000(ChromeDriverService.java:32)
at org.openqa.selenium.chrome.ChromeDriverService$Builder.findDefaultExecutable(ChromeDriverService.java:137)
at org.openqa.selenium.remote.service.DriverService$Builder.build(DriverService.java:296)
at org.openqa.selenium.chrome.ChromeDriverService.createDefaultService(ChromeDriverService.java:88)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:148)
at org.apache.spark.ui.ChromeUISeleniumSuite.beforeAll(ChromeUISeleniumSuite.scala:37)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
```
Hadoop 3.2 / Hive 2.3 / JDK11 is also failing together consistently.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3-jdk-11/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633155899
Since it's a weekend, can we simple revert this first to recover the master branch first? Then, we can make a PR with `[test-maven]` title to make it sure at PR stage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633157461
You can revert directly since this commit is pretty new.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-630629927
**[Test build #122835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122835/testReport)** for PR 28578 at commit [`e701dc9`](https://github.com/apache/spark/commit/e701dc99048e63a4a8472d16eb70b2a41223ca45).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #28578:
URL: https://github.com/apache/spark/pull/28578#discussion_r427076227
##########
File path: dev/run-tests.py
##########
@@ -606,6 +609,8 @@ def main():
print("[info] Found the following changed modules:",
", ".join(x.name for x in changed_modules))
+ excluded_tags.extend(always_excluded_tags)
Review comment:
For environments where Chrome and Chrome driver is not installed, including CI environment, tests tagged as `ChromeUITest` is filtered out when tests are run through `run-tests.py`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633154213
Thanks for taking a look at that. Could you make a follow-up PR, @sarutak ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633156114
Agree with the idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-630731680
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122835/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-630630582
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633156114
Agree. I'll make a revert PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-630731052
**[Test build #122835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122835/testReport)** for PR 28578 at commit [`e701dc9`](https://github.com/apache/spark/commit/e701dc99048e63a4a8472d16eb70b2a41223ca45).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-630731666
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-632278408
@sarutak seems there is code conflict after https://github.com/apache/spark/pull/28585 is merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633155732
After investigating more, it seems to break all Maven jobs. The following `Hadoop 2.7` jobs are broken too.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-hive-1.2/
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-hive-2.3/
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-hive-2.3-jdk-11/
In short, this PR only passes `SBT` environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-631996385
cc @gengliangwang FYI. Looks fine to me too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633154172
Ah, some profiles directly run tests without `run-tests.py`.
It might be better to disable the new suite by default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-631209414
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-631504513
Seems OK to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-633153398
Hi, All.
This seems to break `Hadoop 3.2/Hive 2.3` profile Jenkins job in master branch.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3/472/
...
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3/480/
```
ChromeUISeleniumSuite:
org.apache.spark.ui.ChromeUISeleniumSuite *** ABORTED ***
java.lang.IllegalStateException: The driver executable does not exist: /home/jenkins/workspace/spark-master-test-maven-hadoop-3.2-hive-2.3@2/core/null
at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
at org.openqa.selenium.remote.service.DriverService.checkExecutable(DriverService.java:121)
at org.openqa.selenium.remote.service.DriverService.findExecutable(DriverService.java:116)
at org.openqa.selenium.chrome.ChromeDriverService.access$000(ChromeDriverService.java:32)
at org.openqa.selenium.chrome.ChromeDriverService$Builder.findDefaultExecutable(ChromeDriverService.java:137)
at org.openqa.selenium.remote.service.DriverService$Builder.build(DriverService.java:296)
at org.openqa.selenium.chrome.ChromeDriverService.createDefaultService(ChromeDriverService.java:88)
at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:148)
at org.apache.spark.ui.ChromeUISeleniumSuite.beforeAll(ChromeUISeleniumSuite.scala:37)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28578: [SPARK-31756][WEBUI] Add real headless browser support for UI test
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28578:
URL: https://github.com/apache/spark/pull/28578#issuecomment-631154654
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org