[jira] [Commented] (HIVE-17111) TestSparkCliDriver does not use LocalHiveSparkClient

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180154#comment-16180154 ] 

Hive QA commented on HIVE-17111:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12888976/HIVE-17111.1.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11063 tests executed
*Failed tests:*
{noformat}
TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[spark_local_queries] (batchId=64)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=170)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6982/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6982/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6982/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12888976 - PreCommit-HIVE-Build

> TestSparkCliDriver does not use LocalHiveSparkClient
> ----------------------------------------------------
>
>                 Key: HIVE-17111
>                 URL: https://issues.apache.org/jira/browse/HIVE-17111
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-17111.1.patch
>
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but the HoS still uses decides to use the RemoteHiveSparkClient rather than the LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
>     if (master.equals("local") || master.startsWith("local[")) {
>       // With local spark context, all user sessions share the same spark context.
>       return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
>     } else {
>       return new RemoteHiveSparkClient(hiveconf, sparkConf);
>     }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and sees that it doesn't start with {{local[}} and then decides to use the RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed up some of the tests, and also makes qtests easier to debug since everything will now be run in the same process.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)