You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by lu...@sina.com on 2015/07/16 05:53:34 UTC
HiBench test for hadoop/hive/spark cluster
Hi all
when I am running my HiBench in my spark/Hadoop/Hive cluster. I found there is always a failure in my aggregation test. I doubt this problem maybe some issue relative with my hive settings? attaches are my config file and log file . Any idea to solve this issue?
my Spark cluster is a one node standalone cluster :hadoop:2.7spark:1.3hive:1.2.1
here is the log:Prepare aggregation ...
Exec script: /opt/HiBench-master/workloads/aggregation/prepare/prepare.sh
Parsing conf: /opt/HiBench-master/conf/00-default-properties.conf
Parsing conf: /opt/HiBench-master/conf/10-data-scale-profile.conf
Parsing conf: /opt/HiBench-master/conf/99-user_defined_properties.conf
Parsing conf: /opt/HiBench-master/workloads/aggregation/conf/00-aggregation-default.conf
Parsing conf: /opt/HiBench-master/workloads/aggregation/conf/10-aggregation-userdefine.conf
start HadoopPrepareAggregation bench
hdfs rm -r: /usr/lib/hadoop/bin/hadoop --config /usr/lib/hadoop/etc/hadoop fs -rm -r -skipTrash hdfs://spark-study:9000/HiBench/Aggregation/Input
15/07/16 11:16:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Deleted hdfs://spark-study:9000/HiBench/Aggregation/Input
Pages:120000, USERVISITS:1000000
Submit MapReduce Job: /usr/lib/hadoop/bin/hadoop --config /usr/lib/hadoop/etc/hadoop jar /opt/HiBench-master/src/autogen/target/autogen-4.0-SNAPSHOT-jar-with-dependencies.jar HiBench.DataGen -t hive -b hdfs://spark-study:9000/HiBench/Aggregation -n Input -m 12 -r 6 -p 120000 -v 1000000 -o sequence
15/07/16 11:16:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
sfubmzperrrbupqoq
15/07/16 11:16:38 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/07/16 11:16:40 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:42 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:43 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:43 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:44 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:45 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:45 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:46 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:46 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:48 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:50 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 0, 0 erros, 0 missed
15/07/16 11:16:50 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:51 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 1, 0 erros, 0 missed
15/07/16 11:16:51 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:51 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 2, 0 erros, 0 missed
15/07/16 11:16:52 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:52 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 3, 0 erros, 0 missed
15/07/16 11:16:52 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:53 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 4, 0 erros, 0 missed
15/07/16 11:16:53 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:54 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 5, 0 erros, 0 missed
15/07/16 11:16:55 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
java.io.FileNotFoundException: file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or directory)
java.io.FileNotFoundException: file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or directory)
java.io.FileNotFoundException: file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or directory)
java.io.FileNotFoundException: file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or directory)
15/07/16 11:17:01 WARN mapred.LocalJobRunner: job_local563760878_0002
java.lang.Exception: java.lang.NullPointerException
Caused by: java.lang.NullPointerException
Exception in thread "main" java.io.IOException: Job failed!
15/07/16 11:17:02 INFO mapred.LocalJobRunner: hdfs://spark-study:9000/HiBench/Aggregation/temp/dummy:7+2 > map
ERROR: Hadoop job /opt/HiBench-master/src/autogen/target/autogen-4.0-SNAPSHOT-jar-with-dependencies.jar HiBench.DataGen failed to run successfully.
Hint: You can goto /opt/HiBench-master/report/aggregation/prepare/conf/../bench.log to check for detailed log.
Opening log tail for you:
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at HiBench.DataGen.main(DataGen.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
15/07/16 11:17:02 INFO mapred.LocalJobRunner: hdfs://spark-study:9000/HiBench/Aggregation/temp/dummy:7+2 > map
ERROR: aggregation prepare failed!
--------------------------------
Thanks&Best regards!
San.Luo
Re: HiBench test for hadoop/hive/spark cluster
Posted by Ted Yu <yu...@gmail.com>.
>From log file:
15/07/16 11:16:56 INFO mapred.LocalDistributedCacheManager: Creating
symlink: /tmp/hadoop-root/mapred/local/1437016615898/user_agents <-
/opt/HiBench-master/user_agents
15/07/16 11:16:56 INFO mapred.LocalDistributedCacheManager: Localized
hdfs://spark-study:9000/HiBench/Aggregation/temp/user_agents as
file:/tmp/hadoop-root/mapred/local/ 1437016615898/user_agents
...
java.io.FileNotFoundException:
file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file
or directory)
at java.io.FileInputStream.open(Native Method)
However, FileNotFoundException didn't happen to other localized files, such
as country_codes
FYI
On Wed, Jul 15, 2015 at 8:53 PM, <lu...@sina.com> wrote:
> Hi all
>
> when I am running my HiBench in my spark/Hadoop/Hive cluster. I
> found there is always a failure in my aggregation test. I doubt this
> problem maybe some issue relative with my hive settings? attaches are my
> config file and log file .
>
> Any idea to solve this issue?
>
> my Spark cluster is a one node standalone cluster :
>
> hadoop:2.7
>
> spark:1.3
>
> hive:1.2.1
>
> here is the log:
>
> Prepare aggregation ...
> Exec script: /opt/HiBench-master/workloads/aggregation/prepare/prepare.sh
> Parsing conf: /opt/HiBench-master/conf/00-default-properties.conf
> Parsing conf: /opt/HiBench-master/conf/10-data-scale-profile.conf
> Parsing conf: /opt/HiBench-master/conf/99-user_defined_properties.conf
> Parsing conf:
> /opt/HiBench-master/workloads/aggregation/conf/00-aggregation-default.conf
> Parsing conf:
> /opt/HiBench-master/workloads/aggregation/conf/10-aggregation-userdefine.conf
> start HadoopPrepareAggregation bench
> hdfs rm -r: /usr/lib/hadoop/bin/hadoop --config /usr/lib/hadoop/etc/hadoop
> fs -rm -r -skipTrash hdfs://spark-study:9000/HiBench/Aggregation/Input
> 15/07/16 11:16:26 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Deleted hdfs://spark-study:9000/HiBench/Aggregation/Input
> Pages:120000, USERVISITS:1000000
> Submit MapReduce Job: /usr/lib/hadoop/bin/hadoop --config
> /usr/lib/hadoop/etc/hadoop jar
> /opt/HiBench-master/src/autogen/target/autogen-4.0-SNAPSHOT-jar-with-dependencies.jar
> HiBench.DataGen -t hive -b hdfs://spark-study:9000/HiBench/Aggregation -n
> Input -m 12 -r 6 -p 120000 -v 1000000 -o sequence
> 15/07/16 11:16:34 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> sfubmzperrrbupqoq
> 15/07/16 11:16:38 WARN mapreduce.JobResourceUploader: Hadoop command-line
> option parsing not performed. Implement the Tool interface and execute your
> application with ToolRunner to remedy this.
> 15/07/16 11:16:40 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:42 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:43 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:43 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:44 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:45 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:45 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:46 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:46 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
> 15/07/16 11:16:48 INFO reduce.EventFetcher: EventFetcher is interrupted..
> Returning
> 15/07/16 11:16:50 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 0, 0
> erros, 0 missed
> 15/07/16 11:16:50 INFO reduce.EventFetcher: EventFetcher is interrupted..
> Returning
> 15/07/16 11:16:51 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 1, 0
> erros, 0 missed
> 15/07/16 11:16:51 INFO reduce.EventFetcher: EventFetcher is interrupted..
> Returning
> 15/07/16 11:16:51 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 2, 0
> erros, 0 missed
> 15/07/16 11:16:52 INFO reduce.EventFetcher: EventFetcher is interrupted..
> Returning
> 15/07/16 11:16:52 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 3, 0
> erros, 0 missed
> 15/07/16 11:16:52 INFO reduce.EventFetcher: EventFetcher is interrupted..
> Returning
> 15/07/16 11:16:53 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 4, 0
> erros, 0 missed
> 15/07/16 11:16:53 INFO reduce.EventFetcher: EventFetcher is interrupted..
> Returning
> 15/07/16 11:16:54 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 5, 0
> erros, 0 missed
> 15/07/16 11:16:55 WARN mapreduce.JobResourceUploader: Hadoop command-line
> option parsing not performed. Implement the Tool interface and execute your
> application with ToolRunner to remedy this.
> java.io.FileNotFoundException:
> file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file
> or directory)
> java.io.FileNotFoundException:
> file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file
> or directory)
> java.io.FileNotFoundException:
> file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file
> or directory)
> java.io.FileNotFoundException:
> file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file
> or directory)
> 15/07/16 11:17:01 WARN mapred.LocalJobRunner: job_local563760878_0002
> java.lang.Exception: java.lang.NullPointerException
> Caused by: java.lang.NullPointerException
> Exception in thread "main" java.io.IOException: Job failed!
> 15/07/16 11:17:02 INFO mapred.LocalJobRunner:
> hdfs://spark-study:9000/HiBench/Aggregation/temp/dummy:7+2 > map
> ERROR: Hadoop job
> /opt/HiBench-master/src/autogen/target/autogen-4.0-SNAPSHOT-jar-with-dependencies.jar
> HiBench.DataGen failed to run successfully.
> Hint: You can goto
> /opt/HiBench-master/report/aggregation/prepare/conf/../bench.log to check
> for detailed log.
> Opening log tail for you:
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at HiBench.DataGen.main(DataGen.java:45)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> 15/07/16 11:17:02 INFO mapred.LocalJobRunner:
> hdfs://spark-study:9000/HiBench/Aggregation/temp/dummy:7+2 > map
>
> ERROR: aggregation prepare failed!
>
> --------------------------------
>
> Thanks&Best regards!
> San.Luo
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>