You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Peter Vary (JIRA)" <ji...@apache.org> on 2017/08/08 16:20:00 UTC
[jira] [Commented] (HIVE-17270) Qtest results show wrong number of
executors
[ https://issues.apache.org/jira/browse/HIVE-17270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118561#comment-16118561 ]
Peter Vary commented on HIVE-17270:
-----------------------------------
Also the number of reducers shown in the golden files are 2, and based on the configuration I think it should be 4. See for example:
https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_2.q.out#L197
I think this has to has some connection with the creation and - in case of the config change - the recreation of the RpcServer, which for the first time.
I will dig further into it, but any pointers would be nice [~stakiar], or [~xuefuz].
Thanks,
Peter
> Qtest results show wrong number of executors
> --------------------------------------------
>
> Key: HIVE-17270
> URL: https://issues.apache.org/jira/browse/HIVE-17270
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Affects Versions: 3.0.0
> Reporter: Peter Vary
> Assignee: Peter Vary
>
> The hive-site.xml shows, that the TestMiniSparkOnYarnCliDriver uses 2 cores, and 2 executor instances to run the queries. See: https://github.com/apache/hive/blob/master/data/conf/spark/yarn-client/hive-site.xml#L233
> When reading the log files for the query tests, I see the following:
> {code}
> 2017-08-08T07:41:03,315 INFO [0381325d-2c8c-46fb-ab51-423defaddd84 main] session.SparkSession: Spark cluster current has executors: 1, total cores: 2, memory per executor: 512M, memoryFraction: 0.4
> {code}
> See: http://104.198.109.242/logs/PreCommit-HIVE-Build-6299/succeeded/171-TestMiniSparkOnYarnCliDriver-insert_overwrite_directory2.q-scriptfile1.q-vector_outer_join0.q-and-17-more/logs/hive.log
> When running the tests against a real cluster, I found that running an explain query for the first time I see 1 executor, but running it for the second time I see 2 executors.
> Also setting some spark configuration on the cluster resets this behavior. For the first time I will see 1 executor, and for the second time I will see 2 executors again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)