You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/05 12:44:18 UTC
[GitHub] [spark] yaooqinn opened a new pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all ThriftServer related tests
yaooqinn opened a new pull request #28738:
URL: https://github.com/apache/spark/pull/28738
### What changes were proposed in this pull request?
This PR apply `SharedThriftServer` trait to all ThriftServer related tests including
```
HiveThriftBinaryServerSuite
HiveThriftCleanUpScratchDirSuite
HiveThriftHttpServerSuite
JdbcConnectionUriSuite
SingleSessionSuite
SparkMetadataOperationSuite
SparkThriftServerProtocolVersionsSuite
UISeleniumSuite
```
`SharedThriftServer` can use a random free port to reduce the flakiness of these tests.
This PR also re-enabled the `UISeleniumSuite` which has been ignored since 2.0.
### Why are the changes needed?
1. reduce the flakiness of tests.
2. improve test coverage
3. simplification
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
no
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
existing tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn edited a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
yaooqinn edited a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639524059
> Nice cleanup of the Thriftserver tests!
> I think it still has some merit to test the Thriftserver started with the start-thriftserver.sh script, so I would not remove that old infra completely, but maybe leave some basic suite for that?
`start-thriftserver.sh` script with port retry is flaky for testing, maybe we can dockerize some of them with integration test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639484275
**[Test build #123567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123567/testReport)** for PR 28738 at commit [`9812e50`](https://github.com/apache/spark/commit/9812e50031796672829cfcc2ccf77de0c42bf6b9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639458448
**[Test build #123566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123566/testReport)** for PR 28738 at commit [`9fde365`](https://github.com/apache/spark/commit/9fde36540f2047619f76fa71b4cc64411485a41d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639661844
**[Test build #123571 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123571/testReport)** for PR 28738 at commit [`70d0ab3`](https://github.com/apache/spark/commit/70d0ab383ae35896f9624a54c1289558d33dcc6e).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639614211
**[Test build #123571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123571/testReport)** for PR 28738 at commit [`70d0ab3`](https://github.com/apache/spark/commit/70d0ab383ae35896f9624a54c1289558d33dcc6e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-691752392
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639614211
**[Test build #123571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123571/testReport)** for PR 28738 at commit [`70d0ab3`](https://github.com/apache/spark/commit/70d0ab383ae35896f9624a54c1289558d33dcc6e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639661900
Build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639458448
**[Test build #123566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123566/testReport)** for PR 28738 at commit [`9fde365`](https://github.com/apache/spark/commit/9fde36540f2047619f76fa71b4cc64411485a41d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639524059
> Nice cleanup of the Thriftserver tests!
> I think it still has some merit to test the Thriftserver started with the start-thriftserver.sh script, so I would not remove that old infra completely, but maybe leave some basic suite for that?
`start-thriftserver.sh` script with port retry is flaky for testing, maybe we can dockerize them with integration test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639504526
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639458982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #28738:
URL: https://github.com/apache/spark/pull/28738
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
juliuszsompolski commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639499914
Nice cleanup of the Thriftserver tests!
I think it still has some merit to test the Thriftserver started with the start-thriftserver.sh script, so I would not remove that old infra completely, but maybe leave some basic suite for that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639484275
**[Test build #123567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123567/testReport)** for PR 28738 at commit [`9812e50`](https://github.com/apache/spark/commit/9812e50031796672829cfcc2ccf77de0c42bf6b9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639543260
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123567/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639614872
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639504526
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639661900
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639504365
**[Test build #123566 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123566/testReport)** for PR 28738 at commit [`9fde365`](https://github.com/apache/spark/commit/9fde36540f2047619f76fa71b4cc64411485a41d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HiveThriftBinaryServerSuite extends SharedThriftServer `
* `class SingleSessionSuite extends SharedThriftServer `
* `class HiveThriftCleanUpScratchDirSuite extends SharedThriftServer `
* `class HiveThriftHttpServerSuite extends SharedThriftServer `
* `class JdbcConnectionUriSuite extends SharedThriftServer `
* `class SparkMetadataOperationSuite extends SharedThriftServer `
* `class SparkThriftServerProtocolVersionsSuite extends SharedThriftServer `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639543220
**[Test build #123567 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123567/testReport)** for PR 28738 at commit [`9812e50`](https://github.com/apache/spark/commit/9812e50031796672829cfcc2ccf77de0c42bf6b9).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* `class HiveThriftBinaryServerSuite extends JdbcConnectionSuite `
* `class HiveThriftHttpServerSuite extends JdbcConnectionSuite `
* `trait JdbcConnectionSuite extends SharedThriftServer `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639614872
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639458982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639543247
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639504561
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123566/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639661916
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123571/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639484906
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639543247
Build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] juliuszsompolski commented on a change in pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
juliuszsompolski commented on a change in pull request #28738:
URL: https://github.com/apache/spark/pull/28738#discussion_r435990515
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
##########
@@ -1006,338 +417,12 @@ class HiveThriftCleanUpScratchDirSuite extends HiveThriftJdbcTest{
}
}
- override protected def afterAll(): Unit = {
+ override def afterAll(): Unit = {
Utils.deleteRecursively(tempScratchDir)
super.afterAll()
}
}
-class HiveThriftHttpServerSuite extends HiveThriftJdbcTest {
- override def mode: ServerMode.Value = ServerMode.http
-
- test("JDBC query execution") {
- withJdbcStatement("test") { statement =>
- val queries = Seq(
- "SET spark.sql.shuffle.partitions=3",
- "CREATE TABLE test(key INT, val STRING) USING hive",
- s"LOAD DATA LOCAL INPATH '${TestData.smallKv}' OVERWRITE INTO TABLE test",
- "CACHE TABLE test")
-
- queries.foreach(statement.execute)
-
- assertResult(5, "Row count mismatch") {
- val resultSet = statement.executeQuery("SELECT COUNT(*) FROM test")
- resultSet.next()
- resultSet.getInt(1)
- }
- }
- }
-
- test("Checks Hive version") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SET spark.sql.hive.version")
- resultSet.next()
- assert(resultSet.getString(1) === "spark.sql.hive.version")
- assert(resultSet.getString(2) === HiveUtils.builtinHiveVersion)
- }
- }
-
- test("SPARK-24829 Checks cast as float") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT CAST('4.56' AS FLOAT)")
- resultSet.next()
- assert(resultSet.getString(1) === "4.56")
- }
- }
-}
-
object ServerMode extends Enumeration {
val binary, http = Value
}
-
-abstract class HiveThriftJdbcTest extends HiveThriftServer2Test {
- Utils.classForName(classOf[HiveDriver].getCanonicalName)
-
- private def jdbcUri = if (mode == ServerMode.http) {
- s"""jdbc:hive2://localhost:$serverPort/
- |default?
- |hive.server2.transport.mode=http;
- |hive.server2.thrift.http.path=cliservice;
- |${hiveConfList}#${hiveVarList}
- """.stripMargin.split("\n").mkString.trim
- } else {
- s"jdbc:hive2://localhost:$serverPort/?${hiveConfList}#${hiveVarList}"
- }
-
- def withMultipleConnectionJdbcStatement(tableNames: String*)(fs: (Statement => Unit)*): Unit = {
- val user = System.getProperty("user.name")
- val connections = fs.map { _ => DriverManager.getConnection(jdbcUri, user, "") }
- val statements = connections.map(_.createStatement())
-
- try {
- statements.zip(fs).foreach { case (s, f) => f(s) }
- } finally {
- tableNames.foreach { name =>
- // TODO: Need a better way to drop the view.
- if (name.toUpperCase(Locale.ROOT).startsWith("VIEW")) {
- statements(0).execute(s"DROP VIEW IF EXISTS $name")
- } else {
- statements(0).execute(s"DROP TABLE IF EXISTS $name")
- }
- }
- statements.foreach(_.close())
- connections.foreach(_.close())
- }
- }
-
- def withDatabase(dbNames: String*)(fs: (Statement => Unit)*): Unit = {
- val user = System.getProperty("user.name")
- val connections = fs.map { _ => DriverManager.getConnection(jdbcUri, user, "") }
- val statements = connections.map(_.createStatement())
-
- try {
- statements.zip(fs).foreach { case (s, f) => f(s) }
- } finally {
- dbNames.foreach { name =>
- statements(0).execute(s"DROP DATABASE IF EXISTS $name")
- }
- statements.foreach(_.close())
- connections.foreach(_.close())
- }
- }
-
- def withJdbcStatement(tableNames: String*)(f: Statement => Unit): Unit = {
- withMultipleConnectionJdbcStatement(tableNames: _*)(f)
- }
-}
-
-abstract class HiveThriftServer2Test extends SparkFunSuite with BeforeAndAfterAll with Logging {
- def mode: ServerMode.Value
-
- private val CLASS_NAME = HiveThriftServer2.getClass.getCanonicalName.stripSuffix("$")
- private val LOG_FILE_MARK = s"starting $CLASS_NAME, logging to "
-
- protected val startScript = "../../sbin/start-thriftserver.sh".split("/").mkString(File.separator)
- protected val stopScript = "../../sbin/stop-thriftserver.sh".split("/").mkString(File.separator)
-
- private var listeningPort: Int = _
- protected def serverPort: Int = listeningPort
-
- protected val hiveConfList = "a=avalue;b=bvalue"
- protected val hiveVarList = "c=cvalue;d=dvalue"
- protected def user = System.getProperty("user.name")
-
- protected var warehousePath: File = _
- protected var metastorePath: File = _
- protected def metastoreJdbcUri = s"jdbc:derby:;databaseName=$metastorePath;create=true"
-
- private val pidDir: File = Utils.createTempDir(namePrefix = "thriftserver-pid")
- protected var logPath: File = _
- protected var operationLogPath: File = _
- protected var lScratchDir: File = _
- private var logTailingProcess: Process = _
- private var diagnosisBuffer: ArrayBuffer[String] = ArrayBuffer.empty[String]
-
- protected def extraConf: Seq[String] = Nil
-
- protected def serverStartCommand(port: Int) = {
- val portConf = if (mode == ServerMode.binary) {
- ConfVars.HIVE_SERVER2_THRIFT_PORT
- } else {
- ConfVars.HIVE_SERVER2_THRIFT_HTTP_PORT
- }
-
- val driverClassPath = {
- // Writes a temporary log4j.properties and prepend it to driver classpath, so that it
- // overrides all other potential log4j configurations contained in other dependency jar files.
- val tempLog4jConf = Utils.createTempDir().getCanonicalPath
-
- Files.write(
- """log4j.rootCategory=DEBUG, console
- |log4j.appender.console=org.apache.log4j.ConsoleAppender
- |log4j.appender.console.target=System.err
- |log4j.appender.console.layout=org.apache.log4j.PatternLayout
- |log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
- """.stripMargin,
- new File(s"$tempLog4jConf/log4j.properties"),
- StandardCharsets.UTF_8)
-
- tempLog4jConf
- }
-
- s"""$startScript
- | --master local
- | --hiveconf ${ConfVars.METASTORECONNECTURLKEY}=$metastoreJdbcUri
- | --hiveconf ${ConfVars.METASTOREWAREHOUSE}=$warehousePath
- | --hiveconf ${ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST}=localhost
- | --hiveconf ${ConfVars.HIVE_SERVER2_TRANSPORT_MODE}=$mode
- | --hiveconf ${ConfVars.HIVE_SERVER2_LOGGING_OPERATION_LOG_LOCATION}=$operationLogPath
- | --hiveconf ${ConfVars.LOCALSCRATCHDIR}=$lScratchDir
- | --hiveconf $portConf=$port
- | --driver-class-path $driverClassPath
- | --driver-java-options -Dlog4j.debug
- | --conf spark.ui.enabled=false
- | ${extraConf.mkString("\n")}
- """.stripMargin.split("\\s+").toSeq
- }
-
- /**
- * String to scan for when looking for the thrift binary endpoint running.
- * This can change across Hive versions.
- */
- val THRIFT_BINARY_SERVICE_LIVE = "Starting ThriftBinaryCLIService on port"
-
- /**
- * String to scan for when looking for the thrift HTTP endpoint running.
- * This can change across Hive versions.
- */
- val THRIFT_HTTP_SERVICE_LIVE = "Started ThriftHttpCLIService in http"
Review comment:
If this becomes flaky because of the port, you could set port 0, and turn the use of these lines into a regex that would parse the port that got assigned...
##########
File path: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
##########
@@ -1006,338 +417,12 @@ class HiveThriftCleanUpScratchDirSuite extends HiveThriftJdbcTest{
}
}
- override protected def afterAll(): Unit = {
+ override def afterAll(): Unit = {
Utils.deleteRecursively(tempScratchDir)
super.afterAll()
}
}
-class HiveThriftHttpServerSuite extends HiveThriftJdbcTest {
- override def mode: ServerMode.Value = ServerMode.http
-
- test("JDBC query execution") {
- withJdbcStatement("test") { statement =>
- val queries = Seq(
- "SET spark.sql.shuffle.partitions=3",
- "CREATE TABLE test(key INT, val STRING) USING hive",
- s"LOAD DATA LOCAL INPATH '${TestData.smallKv}' OVERWRITE INTO TABLE test",
- "CACHE TABLE test")
-
- queries.foreach(statement.execute)
-
- assertResult(5, "Row count mismatch") {
- val resultSet = statement.executeQuery("SELECT COUNT(*) FROM test")
- resultSet.next()
- resultSet.getInt(1)
- }
- }
- }
-
- test("Checks Hive version") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SET spark.sql.hive.version")
- resultSet.next()
- assert(resultSet.getString(1) === "spark.sql.hive.version")
- assert(resultSet.getString(2) === HiveUtils.builtinHiveVersion)
- }
- }
-
- test("SPARK-24829 Checks cast as float") {
- withJdbcStatement() { statement =>
- val resultSet = statement.executeQuery("SELECT CAST('4.56' AS FLOAT)")
- resultSet.next()
- assert(resultSet.getString(1) === "4.56")
- }
- }
-}
-
object ServerMode extends Enumeration {
val binary, http = Value
}
-
-abstract class HiveThriftJdbcTest extends HiveThriftServer2Test {
- Utils.classForName(classOf[HiveDriver].getCanonicalName)
-
- private def jdbcUri = if (mode == ServerMode.http) {
- s"""jdbc:hive2://localhost:$serverPort/
- |default?
- |hive.server2.transport.mode=http;
- |hive.server2.thrift.http.path=cliservice;
- |${hiveConfList}#${hiveVarList}
- """.stripMargin.split("\n").mkString.trim
- } else {
- s"jdbc:hive2://localhost:$serverPort/?${hiveConfList}#${hiveVarList}"
- }
-
- def withMultipleConnectionJdbcStatement(tableNames: String*)(fs: (Statement => Unit)*): Unit = {
- val user = System.getProperty("user.name")
- val connections = fs.map { _ => DriverManager.getConnection(jdbcUri, user, "") }
- val statements = connections.map(_.createStatement())
-
- try {
- statements.zip(fs).foreach { case (s, f) => f(s) }
- } finally {
- tableNames.foreach { name =>
- // TODO: Need a better way to drop the view.
- if (name.toUpperCase(Locale.ROOT).startsWith("VIEW")) {
- statements(0).execute(s"DROP VIEW IF EXISTS $name")
- } else {
- statements(0).execute(s"DROP TABLE IF EXISTS $name")
- }
- }
- statements.foreach(_.close())
- connections.foreach(_.close())
- }
- }
-
- def withDatabase(dbNames: String*)(fs: (Statement => Unit)*): Unit = {
- val user = System.getProperty("user.name")
- val connections = fs.map { _ => DriverManager.getConnection(jdbcUri, user, "") }
- val statements = connections.map(_.createStatement())
-
- try {
- statements.zip(fs).foreach { case (s, f) => f(s) }
- } finally {
- dbNames.foreach { name =>
- statements(0).execute(s"DROP DATABASE IF EXISTS $name")
- }
- statements.foreach(_.close())
- connections.foreach(_.close())
- }
- }
-
- def withJdbcStatement(tableNames: String*)(f: Statement => Unit): Unit = {
- withMultipleConnectionJdbcStatement(tableNames: _*)(f)
- }
-}
-
-abstract class HiveThriftServer2Test extends SparkFunSuite with BeforeAndAfterAll with Logging {
- def mode: ServerMode.Value
-
- private val CLASS_NAME = HiveThriftServer2.getClass.getCanonicalName.stripSuffix("$")
- private val LOG_FILE_MARK = s"starting $CLASS_NAME, logging to "
-
- protected val startScript = "../../sbin/start-thriftserver.sh".split("/").mkString(File.separator)
- protected val stopScript = "../../sbin/stop-thriftserver.sh".split("/").mkString(File.separator)
-
- private var listeningPort: Int = _
- protected def serverPort: Int = listeningPort
-
- protected val hiveConfList = "a=avalue;b=bvalue"
- protected val hiveVarList = "c=cvalue;d=dvalue"
- protected def user = System.getProperty("user.name")
-
- protected var warehousePath: File = _
- protected var metastorePath: File = _
- protected def metastoreJdbcUri = s"jdbc:derby:;databaseName=$metastorePath;create=true"
-
- private val pidDir: File = Utils.createTempDir(namePrefix = "thriftserver-pid")
- protected var logPath: File = _
- protected var operationLogPath: File = _
- protected var lScratchDir: File = _
- private var logTailingProcess: Process = _
- private var diagnosisBuffer: ArrayBuffer[String] = ArrayBuffer.empty[String]
-
- protected def extraConf: Seq[String] = Nil
-
- protected def serverStartCommand(port: Int) = {
- val portConf = if (mode == ServerMode.binary) {
- ConfVars.HIVE_SERVER2_THRIFT_PORT
- } else {
- ConfVars.HIVE_SERVER2_THRIFT_HTTP_PORT
- }
-
- val driverClassPath = {
- // Writes a temporary log4j.properties and prepend it to driver classpath, so that it
- // overrides all other potential log4j configurations contained in other dependency jar files.
- val tempLog4jConf = Utils.createTempDir().getCanonicalPath
-
- Files.write(
- """log4j.rootCategory=DEBUG, console
- |log4j.appender.console=org.apache.log4j.ConsoleAppender
- |log4j.appender.console.target=System.err
- |log4j.appender.console.layout=org.apache.log4j.PatternLayout
- |log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
- """.stripMargin,
- new File(s"$tempLog4jConf/log4j.properties"),
- StandardCharsets.UTF_8)
-
- tempLog4jConf
- }
-
- s"""$startScript
- | --master local
- | --hiveconf ${ConfVars.METASTORECONNECTURLKEY}=$metastoreJdbcUri
- | --hiveconf ${ConfVars.METASTOREWAREHOUSE}=$warehousePath
- | --hiveconf ${ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST}=localhost
- | --hiveconf ${ConfVars.HIVE_SERVER2_TRANSPORT_MODE}=$mode
- | --hiveconf ${ConfVars.HIVE_SERVER2_LOGGING_OPERATION_LOG_LOCATION}=$operationLogPath
- | --hiveconf ${ConfVars.LOCALSCRATCHDIR}=$lScratchDir
- | --hiveconf $portConf=$port
- | --driver-class-path $driverClassPath
- | --driver-java-options -Dlog4j.debug
- | --conf spark.ui.enabled=false
- | ${extraConf.mkString("\n")}
- """.stripMargin.split("\\s+").toSeq
- }
-
- /**
- * String to scan for when looking for the thrift binary endpoint running.
- * This can change across Hive versions.
- */
- val THRIFT_BINARY_SERVICE_LIVE = "Starting ThriftBinaryCLIService on port"
-
- /**
- * String to scan for when looking for the thrift HTTP endpoint running.
- * This can change across Hive versions.
- */
- val THRIFT_HTTP_SERVICE_LIVE = "Started ThriftHttpCLIService in http"
-
- val SERVER_STARTUP_TIMEOUT = 3.minutes
-
- private def startThriftServer(port: Int, attempt: Int) = {
- warehousePath = Utils.createTempDir()
- warehousePath.delete()
- metastorePath = Utils.createTempDir()
- metastorePath.delete()
- operationLogPath = Utils.createTempDir()
- operationLogPath.delete()
- lScratchDir = Utils.createTempDir()
- lScratchDir.delete()
- logPath = null
- logTailingProcess = null
-
- val command = serverStartCommand(port)
-
- diagnosisBuffer ++=
- s"""
- |### Attempt $attempt ###
- |HiveThriftServer2 command line: $command
- |Listening port: $port
- |System user: $user
- """.stripMargin.split("\n")
-
- logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode, attempt=$attempt")
-
- logPath = {
- val lines = Utils.executeAndGetOutput(
- command = command,
- extraEnvironment = Map(
- // Disables SPARK_TESTING to exclude log4j.properties in test directories.
- "SPARK_TESTING" -> "0",
- // But set SPARK_SQL_TESTING to make spark-class happy.
- "SPARK_SQL_TESTING" -> "1",
- // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift server instance can be
- // started at a time, which is not Jenkins friendly.
- "SPARK_PID_DIR" -> pidDir.getCanonicalPath),
- redirectStderr = true)
-
- logInfo(s"COMMAND: $command")
- logInfo(s"OUTPUT: $lines")
- lines.split("\n").collectFirst {
- case line if line.contains(LOG_FILE_MARK) => new File(line.drop(LOG_FILE_MARK.length))
- }.getOrElse {
- throw new RuntimeException("Failed to find HiveThriftServer2 log file.")
- }
- }
-
- val serverStarted = Promise[Unit]()
-
- // Ensures that the following "tail" command won't fail.
- logPath.createNewFile()
- val successLines = Seq(THRIFT_BINARY_SERVICE_LIVE, THRIFT_HTTP_SERVICE_LIVE)
-
- logTailingProcess = {
- val command = s"/usr/bin/env tail -n +0 -f ${logPath.getCanonicalPath}".split(" ")
- // Using "-n +0" to make sure all lines in the log file are checked.
- val builder = new ProcessBuilder(command: _*)
- val captureOutput = (line: String) => diagnosisBuffer.synchronized {
- diagnosisBuffer += line
-
- successLines.foreach { r =>
- if (line.contains(r)) {
- serverStarted.trySuccess(())
- }
- }
Review comment:
... by parsing for the port here instead of the line.contains.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28738: [SPARK-31914][SQL][Test][test-hive1.2][test-java11] Apply SharedThriftServer to all thrift server related tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28738:
URL: https://github.com/apache/spark/pull/28738#issuecomment-639484906
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org