Review Request 61716: HIVE-17292 - Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores

[Peter Vary <pv...@cloudera.com>]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61716/
-----------------------------------------------------------

Review request for hive, Rui Li and Xuefu Zhang.


Bugs: HIVE-17292
    https://issues.apache.org/jira/browse/HIVE-17292


Repository: hive-git


Description
-------

Currently MiniSparkShim will be able to provide only 1 executor, since the FairScheduler is configured, and the allocation increment is not set so the default 1024 is used. So only 1 executor could fit into the memory restrictions.

Changing Hadoop23Shims.java, so the MiniSparkShim will able to provide the requested 2 executors.

Changing QTestUtil.setSparkSession, so we will wait until every executor is available, not only the 1st.

Changing SparkSessionImpl.getMemoryAndCores, so we use the client provided paralellism in case of local spark.master too.

Regenerating golden files (numReducers, and number of files changed in the explain plans)

The change contains 2 golden file changes (spark_dynamic_partition_pruning_mapjoin_only.q.out, spark_dynamic_partition_pruning.q.out) which are containing other neccessary changes for a green run, so this patch should be regenerated after their corresponding jiras are solved (HIVE-17347, HIVE-17346)


Diffs
-----

  data/conf/spark/standalone/hive-site.xml 989e65e 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 825f826 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java 8224ef9 
  ql/src/test/results/clientpositive/spark/constprog_semijoin.q.out be33189 
  ql/src/test/results/clientpositive/spark/index_bitmap3.q.out 31c921c 
  ql/src/test/results/clientpositive/spark/index_bitmap_auto.q.out e3b205c 
  ql/src/test/results/clientpositive/spark/infer_bucket_sort_map_operators.q.out 356f625 
  ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out bf010e7 
  ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning.q.out d9da762 
  ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_2.q.out 8a794e7 
  ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_mapjoin_only.q.out 564a6a2 
  ql/src/test/results/clientpositive/spark/spark_explain_groupbyshuffle.q.out 9d162c9 
  ql/src/test/results/clientpositive/spark/spark_use_ts_stats_for_mapjoin.q.out f92356e 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ae6f542 


Diff: https://reviews.apache.org/r/61716/diff/1/


Testing
-------


Thanks,

Peter Vary