You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Thomas Friedrich (JIRA)" <ji...@apache.org> on 2014/10/09 00:32:34 UTC
[jira] [Commented] (HIVE-7953) Investigate query failures (2)
[ https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164299#comment-14164299 ]
Thomas Friedrich commented on HIVE-7953:
----------------------------------------
The following 4 tests are from the minimr.query.files list:
temp_table_external
load_hdfs_file_with_space_in_the_name
external_table_with_space_in_location_path
bucket_num_reducers
The 3 tests
temp_table_external
load_hdfs_file_with_space_in_the_name
external_table_with_space_in_location_path
create dirs in HDFS, but TestSparkCliDriver is not using HDFS. That's why the tests fail when creating a dir, for example mkdir -p hdfs:///tmp/temp_table_external
In the non-spark case, these tests are executed with the TestMinimrCliDriver which initializes Hadoop and created a mini-cluster
(see configuration in itests/qtest/pom.xml).
We may need to configure a TestSparkMinimrCliDriver in itests/qtest-spark/pom.xml if we want to run these test cases.
The test bucket_num_reducers doesn't use HDFS, and fails with a different error:
Exception: Number of MapReduce jobs is incorrect expected:<1> but was:<0>
junit.framework.AssertionFailedError: Number of MapReduce jobs is incorrect expected:<1> but was:<0>
at org.apache.hadoop.hive.ql.hooks.VerifyNumReducersHook.run(VerifyNumReducersHook.java:46)
> Investigate query failures (2)
> ------------------------------
>
> Key: HIVE-7953
> URL: https://issues.apache.org/jira/browse/HIVE-7953
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Brock Noland
> Assignee: Thomas Friedrich
>
> I ran all q-file tests and the following failed with an exception:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/
> we don't necessary want to run all these tests as part of the spark tests, but we should understand why they failed with an exception. This JIRA is to look into these failures and document them with one of:
> * New JIRA
> * Covered under existing JIRA
> * More investigation required
> Tests:
> {noformat}
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external 0.33 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers 4.3 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2 11 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name 0.65 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4 4.7 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7 2.8 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2 5.5 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position 1.5 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external 2.4 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6 11 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11 5.1 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8 10 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join 5.4 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part 0.81 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 0.31 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1 0.26 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2 0.73 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3 8.5 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5 0.34 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata 0.93 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer 6.3 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact3 2.4 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact2 0.56 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_partscan_1_23 3.1 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2 4.3 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_15_external_part 3.2 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_16_part_external 2.8 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_17_part_managed 3.4 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_20_part_managed_location 3.3 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_19_00_part_external_location 6.9 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_external_table_with_space_in_location_path
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)