[jira] [Commented] (HIVE-7709) Create SparkReporter [Spark Branch]

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174865#comment-14174865 ] 

Hive QA commented on HIVE-7709:
-------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12675445/HIVE-7709.1-spark.patch

{color:red}ERROR:{color} -1 due to 141 failed/errored test(s), 6769 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_tez_smb_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_add_part_multiple
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_avro_decimal_native
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_create_merge_compressed
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ctas
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_custom_input_output_format
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_date_udf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_enforce_order
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3_map
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3_map_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3_noskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby3_noskew_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby7_map
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby7_map_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby7_map_skew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby7_noskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_cube1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_ppr
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_rollup1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_innerjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input1_limit
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input_part2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_into3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join21
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join24
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join25
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join26
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join27
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_1to1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_casesensitive
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_nullsafe
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_merge1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_merge2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_gby3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_lateral_view
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_move_tasks_share_dependencies
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel_join1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_multi_insert
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_exists
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_lazy
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_udf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_transform1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union25
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union_remove_16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_data_types
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_left_outer_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_part_project
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_mapjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_nested_mapjoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_shufflejoin
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_timestamp_funcs
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/227/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/227/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-227/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 141 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12675445

> Create SparkReporter [Spark Branch]
> -----------------------------------
>
>                 Key: HIVE-7709
>                 URL: https://issues.apache.org/jira/browse/HIVE-7709
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Chengxiang Li
>            Assignee: Chengxiang Li
>              Labels: Spark-M3
>         Attachments: HIVE-7709.1-spark.patch
>
>
> Hive operators use Reporter to collect global information,  with Hive on Spark mode, we need a new implementation of Reporter to collect hive operator level information based on spark specified Counter. This task should depends on HIVE-7551.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)