You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Evans Ye (JIRA)" <ji...@apache.org> on 2016/01/13 18:52:39 UTC
[jira] [Commented] (BIGTOP-2148) generator.SparkDriver outputs
empty data in cluster mode
[ https://issues.apache.org/jira/browse/BIGTOP-2148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096675#comment-15096675 ]
Evans Ye commented on BIGTOP-2148:
----------------------------------
Looks good to me and +1.
[~rnowling] this is a simple fix would you mind to double confirm?
> generator.SparkDriver outputs empty data in cluster mode
> --------------------------------------------------------
>
> Key: BIGTOP-2148
> URL: https://issues.apache.org/jira/browse/BIGTOP-2148
> Project: Bigtop
> Issue Type: Bug
> Components: blueprints
> Reporter: Kengo Seki
> Assignee: Kengo Seki
> Labels: bigpetstore
> Attachments: BIGTOP-2148.00.patch
>
>
> About generator.SparkDriver, bigpetstore-spark/README.md says:
> bq. You will need to change the master if you want to run on a cluster.
> But in reality, if you run it on a cluster, you'll get empty data:
> {code}
> [sekikn@mobile bigpetstore-spark]$ HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/2.7.1/libexec/etc/hadoop spark-submit --master yarn-cluster --class org.apache.bigtop.bigpetstore.spark.generator.SparkDriver build/libs/bigpetstore-spark-1.1.0-SNAPSHOT-all.jar generated_data 10 1000 365.0 345
> (snip)
> 15/11/18 00:12:30 INFO Client: Application report for application_1447772975157_0003 (state: FINISHED)
> 15/11/18 00:12:30 INFO Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: 192.168.0.4
> ApplicationMaster RPC port: 0
> queue: default
> start time: 1447773097856
> final status: SUCCEEDED
> tracking URL: http://mobile.local:8088/proxy/application_1447772975157_0003/
> user: sekikn
> 15/11/18 00:12:30 INFO ShutdownHookManager: Shutdown hook called
> 15/11/18 00:12:30 INFO ShutdownHookManager: Deleting directory /private/var/folders/n2/1bnspz7j4q7100jmh610zd200000gn/T/spark-ccfcde0c-ea95-4361-b2b3-b709a92bee59
> [sekikn@mobile bigpetstore-spark]$ hdfs dfs -ls generated_data/transactions
> 15/11/18 00:13:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Found 3 items
> -rw-r--r-- 3 sekikn supergroup 0 2015-11-18 00:12 generated_data/transactions/_SUCCESS
> -rw-r--r-- 3 sekikn supergroup 0 2015-11-18 00:12 generated_data/transactions/part-00000
> -rw-r--r-- 3 sekikn supergroup 0 2015-11-18 00:12 generated_data/transactions/part-00001
> {code}
> This is because simulationLength is a variable and always -1 in RDD function. It must be a constant or broadcasted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)