You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by lu...@sina.com on 2015/07/16 06:38:29 UTC
回复：Re: HiBench test for hadoop/hive/spark cluster
Hi Ted Thanks for your advice, i found that there is something wrong with "hadoop fs -get" command, 'cause I believe the localization of hdfs://spark-study:9000/HiBench/Aggregation/temp/user_agents to /tmp/hadoop-root/mapred/local/1437016615898/user_agents is a behaviour like "hadoop fs -get" I am forwarding to solve below issue now,:[root@spark-study HiBench-master]# hadoop fs -get /HiBench/Aggregation/temp/user_agents
15/07/16 12:31:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/07/16 12:31:58 WARN hdfs.DFSClient: DFSInputStream has been closed already
Hi Mike: I am new to Hibench, so I just setup a test enviroment of 1 node spark/hadoop cluster to test, no data actually. Because Hibench will autogenerate test data itself.
--------------------------------
Thanks&Best regards!
San.Luo
----- 原始邮件 -----
发件人：Ted Yu <yu...@gmail.com>
收件人：罗辉 <lu...@sina.com>
抄送人："jie.huang" <ji...@intel.com>, user <us...@spark.apache.org>
主题：Re: HiBench test for hadoop/hive/spark cluster
日期：2015年07月16日 12点17分
From log file:
15/07/16 11:16:56 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-root/mapred/local/1437016615898/user_agents <- /opt/HiBench-master/user_agents15/07/16 11:16:56 INFO mapred.LocalDistributedCacheManager: Localized hdfs://spark-study:9000/HiBench/Aggregation/temp/user_agents as file:/tmp/hadoop-root/mapred/local/ 1437016615898/user_agents...java.io.FileNotFoundException: file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or directory) at java.io.FileInputStream.open(Native Method)
However, FileNotFoundException didn't happen to other localized files, such as country_codes
FYI
On Wed, Jul 15, 2015 at 8:53 PM, <lu...@sina.com> wrote:
Hi all
when I am running my HiBench in my spark/Hadoop/Hive cluster. I found there is always a failure in my aggregation test. I doubt this problem maybe some issue relative with my hive settings? attaches are my config file and log file . Any idea to solve this issue?
my Spark cluster is a one node standalone cluster :hadoop:2.7spark:1.3hive:1.2.1
here is the log:Prepare aggregation ...
Exec script: /opt/HiBench-master/workloads/aggregation/prepare/prepare.sh
Parsing conf: /opt/HiBench-master/conf/00-default-properties.conf
Parsing conf: /opt/HiBench-master/conf/10-data-scale-profile.conf
Parsing conf: /opt/HiBench-master/conf/99-user_defined_properties.conf
Parsing conf: /opt/HiBench-master/workloads/aggregation/conf/00-aggregation-default.conf
Parsing conf: /opt/HiBench-master/workloads/aggregation/conf/10-aggregation-userdefine.conf
start HadoopPrepareAggregation bench
hdfs rm -r: /usr/lib/hadoop/bin/hadoop --config /usr/lib/hadoop/etc/hadoop fs -rm -r -skipTrash hdfs://spark-study:9000/HiBench/Aggregation/Input
15/07/16 11:16:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Deleted hdfs://spark-study:9000/HiBench/Aggregation/Input
Pages:120000, USERVISITS:1000000
Submit MapReduce Job: /usr/lib/hadoop/bin/hadoop --config /usr/lib/hadoop/etc/hadoop jar /opt/HiBench-master/src/autogen/target/autogen-4.0-SNAPSHOT-jar-with-dependencies.jar HiBench.DataGen -t hive -b hdfs://spark-study:9000/HiBench/Aggregation -n Input -m 12 -r 6 -p 120000 -v 1000000 -o sequence
15/07/16 11:16:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
sfubmzperrrbupqoq
15/07/16 11:16:38 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/07/16 11:16:40 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:42 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:43 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:43 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:44 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:45 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:45 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:46 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:46 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:47 INFO HiBench.HtmlCore: WARNING: dict empty!!!
15/07/16 11:16:48 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:50 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 0, 0 erros, 0 missed
15/07/16 11:16:50 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:51 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 1, 0 erros, 0 missed
15/07/16 11:16:51 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:51 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 2, 0 erros, 0 missed
15/07/16 11:16:52 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:52 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 3, 0 erros, 0 missed
15/07/16 11:16:52 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:53 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 4, 0 erros, 0 missed
15/07/16 11:16:53 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
15/07/16 11:16:54 INFO HiBench.HiveData$GenerateRankingsReducer: pid: 5, 0 erros, 0 missed
15/07/16 11:16:55 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
java.io.FileNotFoundException: file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or directory)
java.io.FileNotFoundException: file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or directory)
java.io.FileNotFoundException: file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or directory)
java.io.FileNotFoundException: file:/tmp/hadoop-root/mapred/local/1437016615898/user_agents (No such file or directory)
15/07/16 11:17:01 WARN mapred.LocalJobRunner: job_local563760878_0002
java.lang.Exception: java.lang.NullPointerException
Caused by: java.lang.NullPointerException
Exception in thread "main" java.io.IOException: Job failed!
15/07/16 11:17:02 INFO mapred.LocalJobRunner: hdfs://spark-study:9000/HiBench/Aggregation/temp/dummy:7+2 > map
ERROR: Hadoop job /opt/HiBench-master/src/autogen/target/autogen-4.0-SNAPSHOT-jar-with-dependencies.jar HiBench.DataGen failed to run successfully.
Hint: You can goto /opt/HiBench-master/report/aggregation/prepare/conf/../bench.log to check for detailed log.
Opening log tail for you:
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at HiBench.DataGen.main(DataGen.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
15/07/16 11:17:02 INFO mapred.LocalJobRunner: hdfs://spark-study:9000/HiBench/Aggregation/temp/dummy:7+2 > map
ERROR: aggregation prepare failed!
--------------------------------
Thanks&Best regards!
San.Luo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org