[jira] Commented: (HIVE-396) Hive performance benchmarks

["Vasilis Liaskovitis (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805574#action_12805574 ] 

Vasilis Liaskovitis commented on HIVE-396:
------------------------------------------

I am trying to run the hive benchmarks on a small 8-node hadoop cluster, starting from the 2009-07-21 tarball. Iam using hadoop-0.20.1 and hive-0.4.1. I have a few questions, any help is welcome

- datasize: the report states 60GB of data used for the uservisitis table. The default datagen/htmlgen/config.txt uses ~15GB of Uservisits tables per node. I believe the way the generatedata.py script runs, we get one rankings/ uservisits table per node i.e. on an 10 data nodes we 'd get 10tablesx15GB=150GB of data. Do the benchmarks in the report use all of the generated data i.e. UserVistis_0, UserVists_1,..., UserVistis_10 across the 10 data nodes? Or a subset of these? Same question for the rankings table.
  
- I assume that the hadoop-site.xml and hive-site.xml files under confs/ (from the 2009-07-21 tarball) accurately reflect the hadoop/hive configuration used to produce the results in the report. Let me know if that's not the case, I ve been using these configs to run the benchmarks.

- Regarding query4 (join) specifically ,  the reduce phase of the first job (main job with ) shows a strange behaviour. Looking at the jobtracker GUI, out of 60 reducers, 59 reducers finish relatively fast (~1 min), but one reducer  task takes much longer (~19min). Does that point to a specific hadoop/ hive misconfiguration? No other hadoop job that I 've run exhibits this. Is this intuitive behaviour given a join operation?


> Hive performance benchmarks
> ---------------------------
>
>                 Key: HIVE-396
>                 URL: https://issues.apache.org/jira/browse/HIVE-396
>             Project: Hadoop Hive
>          Issue Type: New Feature
>            Reporter: Zheng Shao
>            Assignee: Yuntao Jia
>         Attachments: AlansMRcode.tgz, hive_benchmark_2009-06-18.pdf, hive_benchmark_2009-06-18.tar.gz, hive_benchmark_2009-07-12.pdf, hive_benchmark_2009-07-21.tar.gz
>
>
> We need some performance benchmark to measure and track the performance improvements of Hive.
> Some references:
> PIG performance benchmarks PIG-200
> PigMix: http://wiki.apache.org/pig/PigMix

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.