You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by chengxiang li <ch...@intel.com> on 2015/01/26 07:37:54 UTC
Review Request 30264: HIVE-9221 enable unit test for mini Spark on
YARN cluster[Spark Branch]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30264/
-----------------------------------------------------------
Review request for hive, Szehon Ho and Xuefu Zhang.
Bugs: HIVE-9211
https://issues.apache.org/jira/browse/HIVE-9211
Repository: hive-git
Description
-------
MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on miniYarnCluster on yarn-client mode, all qfiles in minimr.query.files are enabled in this unit test except 3 qfile: bucket_num_reducers.q, bucket_num_reducers2.q, udf_using.q, which is not supported in HoS.
Diffs
-----
data/conf/spark/hive-site.xml 016f568
data/conf/spark/standalone/hive-site.xml PRE-CREATION
data/conf/spark/yarn-client/hive-site.xml PRE-CREATION
itests/pom.xml e1e88f6
itests/qtest-spark/pom.xml d12fad5
itests/src/test/resources/testconfiguration.properties f583aaf
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 41a2ab7
ql/src/test/results/clientpositive/miniSparkOnYarn/auto_sortmerge_join_16.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucket4.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucket5.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucket6.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucketizedhiveinputformat.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin6.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin7.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/constprog_partitioner.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/disable_merge_for_bucketing.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/empty_dir_in_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/external_table_with_space_in_location_path.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/file_with_header_footer.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/groupby1.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/groupby2.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/import_exported_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap3.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap_auto.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_bucketed_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_dyn_part.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_map_operators.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_merge.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_num_buckets.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_reducers_power_two.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/input16_cc.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/join1.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/leftsemijoin_mr.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/list_bucket_dml_10.q.java1.7.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/load_fs2.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/load_hdfs_file_with_space_in_the_name.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/parallel_orderby.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/quotedid_smb.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/reduce_deduplicate.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/remote_script.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/root_dir_external_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority2.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/scriptfile1.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/smb_mapjoin_8.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter_partitioned.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/temp_table_external.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/truncate_column_buckets.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/uber_reduce.q.out PRE-CREATION
shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b17f465
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java a61c3ac
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java PRE-CREATION
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c
spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db
Diff: https://reviews.apache.org/r/30264/diff/
Testing
-------
Thanks,
chengxiang li
Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark
on YARN cluster[Spark Branch]
Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30264/#review69728
-----------------------------------------------------------
Looks good, glad we can mostly re-use miniYarnCluster. Some minor comments below.
I also agree with Xuefu we dont need more golden files by having some output directory. I'm also ok with not running entire set of spark tests with every run, and just running the list we got from minimr.query.files, it is up to you guys. Only note is, once we check this in, committer will also make some edits to the build machine files to run these in proper batches, o/w they will run the entire TestMiniSparkOnYarnCliDriver suite with one batch.
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java
<https://reviews.apache.org/r/30264/#comment114485>
Need the apache license.
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java
<https://reviews.apache.org/r/30264/#comment114486>
Need a comment.
- Szehon Ho
On Jan. 26, 2015, 6:37 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30264/
> -----------------------------------------------------------
>
> (Updated Jan. 26, 2015, 6:37 a.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-9211
> https://issues.apache.org/jira/browse/HIVE-9211
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on miniYarnCluster on yarn-client mode, all qfiles in minimr.query.files are enabled in this unit test except 3 qfile: bucket_num_reducers.q, bucket_num_reducers2.q, udf_using.q, which is not supported in HoS.
>
>
> Diffs
> -----
>
> data/conf/spark/hive-site.xml 016f568
> data/conf/spark/standalone/hive-site.xml PRE-CREATION
> data/conf/spark/yarn-client/hive-site.xml PRE-CREATION
> itests/pom.xml e1e88f6
> itests/qtest-spark/pom.xml d12fad5
> itests/src/test/resources/testconfiguration.properties f583aaf
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 41a2ab7
> ql/src/test/results/clientpositive/miniSparkOnYarn/auto_sortmerge_join_16.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucket4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucket5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucket6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucketizedhiveinputformat.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/constprog_partitioner.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/disable_merge_for_bucketing.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/empty_dir_in_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/external_table_with_space_in_location_path.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/file_with_header_footer.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/groupby1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/groupby2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/import_exported_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_bucketed_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_dyn_part.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_map_operators.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_merge.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_num_buckets.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_reducers_power_two.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/input16_cc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/join1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/leftsemijoin_mr.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/list_bucket_dml_10.q.java1.7.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/load_fs2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/load_hdfs_file_with_space_in_the_name.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/parallel_orderby.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/quotedid_smb.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/reduce_deduplicate.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/remote_script.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/root_dir_external_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/scriptfile1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/smb_mapjoin_8.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/temp_table_external.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/truncate_column_buckets.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/uber_reduce.q.out PRE-CREATION
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b17f465
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java a61c3ac
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java PRE-CREATION
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c
> spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db
>
> Diff: https://reviews.apache.org/r/30264/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>
Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark
on YARN cluster[Spark Branch]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30264/#review69744
-----------------------------------------------------------
Ship it!
Ship It!
- Xuefu Zhang
On Jan. 27, 2015, 2:03 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30264/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2015, 2:03 a.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-9211
> https://issues.apache.org/jira/browse/HIVE-9211
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on miniYarnCluster on yarn-client mode, all qfiles in minimr.query.files are enabled in this unit test except 3 qfile: bucket_num_reducers.q, bucket_num_reducers2.q, udf_using.q, which is not supported in HoS.
>
>
> Diffs
> -----
>
> data/conf/spark/hive-site.xml 016f568
> data/conf/spark/standalone/hive-site.xml PRE-CREATION
> data/conf/spark/yarn-client/hive-site.xml PRE-CREATION
> itests/pom.xml e1e88f6
> itests/qtest-spark/pom.xml d12fad5
> itests/src/test/resources/testconfiguration.properties f583aaf
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 41a2ab7
> ql/src/test/results/clientpositive/spark/bucket5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucketizedhiveinputformat.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/empty_dir_in_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/external_table_with_space_in_location_path.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/file_with_header_footer.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/import_exported_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/index_bitmap3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/index_bitmap_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_bucketed_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_dyn_part.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_map_operators.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_merge.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/input16_cc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/load_fs2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/load_hdfs_file_with_space_in_the_name.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/parallel_orderby.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ql_rewrite_gbtoidx.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/quotedid_smb.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/remote_script.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/root_dir_external_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/schemeAuthority.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/schemeAuthority2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/temp_table_external.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/truncate_column_buckets.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/uber_reduce.q.out PRE-CREATION
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b17f465
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java a61c3ac
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java PRE-CREATION
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c
> spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db
>
> Diff: https://reviews.apache.org/r/30264/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>
Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark
on YARN cluster[Spark Branch]
Posted by Szehon Ho <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30264/#review69734
-----------------------------------------------------------
Ship it!
Looks good to me, leave it to Xuefu to review the other comments.
- Szehon Ho
On Jan. 27, 2015, 2:03 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30264/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2015, 2:03 a.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-9211
> https://issues.apache.org/jira/browse/HIVE-9211
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on miniYarnCluster on yarn-client mode, all qfiles in minimr.query.files are enabled in this unit test except 3 qfile: bucket_num_reducers.q, bucket_num_reducers2.q, udf_using.q, which is not supported in HoS.
>
>
> Diffs
> -----
>
> data/conf/spark/hive-site.xml 016f568
> data/conf/spark/standalone/hive-site.xml PRE-CREATION
> data/conf/spark/yarn-client/hive-site.xml PRE-CREATION
> itests/pom.xml e1e88f6
> itests/qtest-spark/pom.xml d12fad5
> itests/src/test/resources/testconfiguration.properties f583aaf
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 41a2ab7
> ql/src/test/results/clientpositive/spark/bucket5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucket6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/bucketizedhiveinputformat.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/empty_dir_in_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/external_table_with_space_in_location_path.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/file_with_header_footer.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/import_exported_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/index_bitmap3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/index_bitmap_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_bucketed_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_dyn_part.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_map_operators.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_merge.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/input16_cc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/load_fs2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/load_hdfs_file_with_space_in_the_name.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/parallel_orderby.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ql_rewrite_gbtoidx.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/quotedid_smb.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/remote_script.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/root_dir_external_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/schemeAuthority.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/schemeAuthority2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/temp_table_external.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/truncate_column_buckets.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/uber_reduce.q.out PRE-CREATION
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b17f465
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java a61c3ac
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java PRE-CREATION
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c
> spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db
>
> Diff: https://reviews.apache.org/r/30264/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>
Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark
on YARN cluster[Spark Branch]
Posted by chengxiang li <ch...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30264/
-----------------------------------------------------------
(Updated Jan. 27, 2015, 2:03 a.m.)
Review request for hive, Szehon Ho and Xuefu Zhang.
Changes
-------
fixed commented issues.
Bugs: HIVE-9211
https://issues.apache.org/jira/browse/HIVE-9211
Repository: hive-git
Description
-------
MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on miniYarnCluster on yarn-client mode, all qfiles in minimr.query.files are enabled in this unit test except 3 qfile: bucket_num_reducers.q, bucket_num_reducers2.q, udf_using.q, which is not supported in HoS.
Diffs (updated)
-----
data/conf/spark/hive-site.xml 016f568
data/conf/spark/standalone/hive-site.xml PRE-CREATION
data/conf/spark/yarn-client/hive-site.xml PRE-CREATION
itests/pom.xml e1e88f6
itests/qtest-spark/pom.xml d12fad5
itests/src/test/resources/testconfiguration.properties f583aaf
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 41a2ab7
ql/src/test/results/clientpositive/spark/bucket5.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/bucket6.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/bucketizedhiveinputformat.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/empty_dir_in_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/external_table_with_space_in_location_path.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/file_with_header_footer.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/import_exported_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/index_bitmap3.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/index_bitmap_auto.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/infer_bucket_sort_bucketed_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/infer_bucket_sort_dyn_part.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/infer_bucket_sort_map_operators.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/infer_bucket_sort_merge.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/infer_bucket_sort_num_buckets.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/input16_cc.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/list_bucket_dml_10.q.java1.7.out PRE-CREATION
ql/src/test/results/clientpositive/spark/load_fs2.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/load_hdfs_file_with_space_in_the_name.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/parallel_orderby.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/ql_rewrite_gbtoidx.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/quotedid_smb.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/reduce_deduplicate.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/remote_script.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/root_dir_external_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/schemeAuthority.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/schemeAuthority2.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/temp_table_external.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/truncate_column_buckets.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/uber_reduce.q.out PRE-CREATION
shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b17f465
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java a61c3ac
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java PRE-CREATION
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c
spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db
Diff: https://reviews.apache.org/r/30264/diff/
Testing
-------
Thanks,
chengxiang li
Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark
on YARN cluster[Spark Branch]
Posted by chengxiang li <ch...@intel.com>.
> On 一月 26, 2015, 10:30 p.m., Xuefu Zhang wrote:
> > data/conf/spark/yarn-client/hive-site.xml, line 225
> > <https://reviews.apache.org/r/30264/diff/1/?file=834064#file834064line225>
> >
> > Only one executor? Maybe 2 will make it more general.
Yes, that make sense.
On 一月 26, 2015, 10:30 p.m., chengxiang li wrote:
> > I'm wondering why we have a new set of .out files?
Every Test*CliDriver has its own output directory, i didn't think much about this previously. With your remind, i think, yes, we could share the golden files with TestSparkCliDriver, as it's golden file should be the same as TestMiniSparkOnYarnCliDriver for each qtest. One thing more to be note here is that, as spark.query.files contains more than 500 qtests, and it take long enough time to run a full unit test for hive now, i didn't enable all spark.query.files qtest for TestMiniSparkOnYarnCliDriver, instead, i enable qtests from minimr,query.files for it, which contains about 50 qtests, it takes about 10 minutes in my own desktop.
- chengxiang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30264/#review69685
-----------------------------------------------------------
On 一月 26, 2015, 6:37 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30264/
> -----------------------------------------------------------
>
> (Updated 一月 26, 2015, 6:37 a.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-9211
> https://issues.apache.org/jira/browse/HIVE-9211
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on miniYarnCluster on yarn-client mode, all qfiles in minimr.query.files are enabled in this unit test except 3 qfile: bucket_num_reducers.q, bucket_num_reducers2.q, udf_using.q, which is not supported in HoS.
>
>
> Diffs
> -----
>
> data/conf/spark/hive-site.xml 016f568
> data/conf/spark/standalone/hive-site.xml PRE-CREATION
> data/conf/spark/yarn-client/hive-site.xml PRE-CREATION
> itests/pom.xml e1e88f6
> itests/qtest-spark/pom.xml d12fad5
> itests/src/test/resources/testconfiguration.properties f583aaf
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 41a2ab7
> ql/src/test/results/clientpositive/miniSparkOnYarn/auto_sortmerge_join_16.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucket4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucket5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucket6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucketizedhiveinputformat.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/constprog_partitioner.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/disable_merge_for_bucketing.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/empty_dir_in_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/external_table_with_space_in_location_path.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/file_with_header_footer.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/groupby1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/groupby2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/import_exported_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_bucketed_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_dyn_part.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_map_operators.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_merge.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_num_buckets.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_reducers_power_two.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/input16_cc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/join1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/leftsemijoin_mr.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/list_bucket_dml_10.q.java1.7.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/load_fs2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/load_hdfs_file_with_space_in_the_name.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/parallel_orderby.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/quotedid_smb.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/reduce_deduplicate.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/remote_script.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/root_dir_external_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/scriptfile1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/smb_mapjoin_8.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/temp_table_external.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/truncate_column_buckets.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/uber_reduce.q.out PRE-CREATION
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b17f465
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java a61c3ac
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java PRE-CREATION
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c
> spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db
>
> Diff: https://reviews.apache.org/r/30264/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>
Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark
on YARN cluster[Spark Branch]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30264/#review69685
-----------------------------------------------------------
data/conf/spark/yarn-client/hive-site.xml
<https://reviews.apache.org/r/30264/#comment114411>
Only one executor? Maybe 2 will make it more general.
I'm wondering why we have a new set of .out files?
- Xuefu Zhang
On Jan. 26, 2015, 6:37 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30264/
> -----------------------------------------------------------
>
> (Updated Jan. 26, 2015, 6:37 a.m.)
>
>
> Review request for hive, Szehon Ho and Xuefu Zhang.
>
>
> Bugs: HIVE-9211
> https://issues.apache.org/jira/browse/HIVE-9211
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on miniYarnCluster on yarn-client mode, all qfiles in minimr.query.files are enabled in this unit test except 3 qfile: bucket_num_reducers.q, bucket_num_reducers2.q, udf_using.q, which is not supported in HoS.
>
>
> Diffs
> -----
>
> data/conf/spark/hive-site.xml 016f568
> data/conf/spark/standalone/hive-site.xml PRE-CREATION
> data/conf/spark/yarn-client/hive-site.xml PRE-CREATION
> itests/pom.xml e1e88f6
> itests/qtest-spark/pom.xml d12fad5
> itests/src/test/resources/testconfiguration.properties f583aaf
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 41a2ab7
> ql/src/test/results/clientpositive/miniSparkOnYarn/auto_sortmerge_join_16.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucket4.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucket5.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucket6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucketizedhiveinputformat.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin6.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin7.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/constprog_partitioner.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/disable_merge_for_bucketing.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/empty_dir_in_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/external_table_with_space_in_location_path.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/file_with_header_footer.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/groupby1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/groupby2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/import_exported_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap3.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap_auto.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_bucketed_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_dyn_part.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_map_operators.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_merge.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_num_buckets.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_reducers_power_two.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/input16_cc.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/join1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/leftsemijoin_mr.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/list_bucket_dml_10.q.java1.7.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/load_fs2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/load_hdfs_file_with_space_in_the_name.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/parallel_orderby.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/quotedid_smb.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/reduce_deduplicate.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/remote_script.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/root_dir_external_table.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/scriptfile1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/smb_mapjoin_8.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter_partitioned.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/temp_table_external.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/truncate_column_buckets.q.out PRE-CREATION
> ql/src/test/results/clientpositive/miniSparkOnYarn/uber_reduce.q.out PRE-CREATION
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b17f465
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java a61c3ac
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java PRE-CREATION
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c
> spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db
>
> Diff: https://reviews.apache.org/r/30264/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>
Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark
on YARN cluster[Spark Branch]
Posted by chengxiang li <ch...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30264/
-----------------------------------------------------------
(Updated Jan. 26, 2015, 6:37 a.m.)
Review request for hive, Szehon Ho and Xuefu Zhang.
Bugs: HIVE-9211
https://issues.apache.org/jira/browse/HIVE-9211
Repository: hive-git
Description
-------
MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on miniYarnCluster on yarn-client mode, all qfiles in minimr.query.files are enabled in this unit test except 3 qfile: bucket_num_reducers.q, bucket_num_reducers2.q, udf_using.q, which is not supported in HoS.
Diffs
-----
data/conf/spark/hive-site.xml 016f568
data/conf/spark/standalone/hive-site.xml PRE-CREATION
data/conf/spark/yarn-client/hive-site.xml PRE-CREATION
itests/pom.xml e1e88f6
itests/qtest-spark/pom.xml d12fad5
itests/src/test/resources/testconfiguration.properties f583aaf
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 41a2ab7
ql/src/test/results/clientpositive/miniSparkOnYarn/auto_sortmerge_join_16.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucket4.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucket5.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucket6.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucketizedhiveinputformat.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin6.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin7.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/constprog_partitioner.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/disable_merge_for_bucketing.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/empty_dir_in_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/external_table_with_space_in_location_path.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/file_with_header_footer.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/groupby1.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/groupby2.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/import_exported_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap3.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap_auto.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_bucketed_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_dyn_part.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_map_operators.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_merge.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_num_buckets.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_reducers_power_two.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/input16_cc.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/join1.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/leftsemijoin_mr.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/list_bucket_dml_10.q.java1.7.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/load_fs2.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/load_hdfs_file_with_space_in_the_name.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/parallel_orderby.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/quotedid_smb.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/reduce_deduplicate.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/remote_script.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/root_dir_external_table.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority2.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/scriptfile1.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/smb_mapjoin_8.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter_partitioned.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/temp_table_external.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/truncate_column_buckets.q.out PRE-CREATION
ql/src/test/results/clientpositive/miniSparkOnYarn/uber_reduce.q.out PRE-CREATION
shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b17f465
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java a61c3ac
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java PRE-CREATION
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c
spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db
Diff: https://reviews.apache.org/r/30264/diff/
Testing
-------
Thanks,
chengxiang li