You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Kengo Seki (JIRA)" <ji...@apache.org> on 2015/11/17 16:31:11 UTC
[jira] [Updated] (BIGTOP-2148) generator.SparkDriver outputs empty
data in cluster mode
[ https://issues.apache.org/jira/browse/BIGTOP-2148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kengo Seki updated BIGTOP-2148:
-------------------------------
Attachment: BIGTOP-2148.00.patch
Attaching a patch. After applying this, generator.SparkDriver generates non-empty output as follows:
{code}
[sekikn@mobile bigpetstore-spark]$ hdfs dfs -rm -r generated_data; HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/2.7.1/libexec/etc/hadoop spark-submit --master yarn-cluster --class org.apache.bigtop.bigpetstore.spark.generator.SparkDriver build/libs/bigpetstore-spark-1.1.0-SNAPSHOT-all.jar generated_data 10 1000 365.0 345
(snip)
15/11/18 00:24:36 INFO Client: Application report for application_1447772975157_0005 (state: FINISHED)
15/11/18 00:24:36 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.0.4
ApplicationMaster RPC port: 0
queue: default
start time: 1447773822522
final status: SUCCEEDED
tracking URL: http://mobile.local:8088/proxy/application_1447772975157_0005/
user: sekikn
15/11/18 00:24:37 INFO ShutdownHookManager: Shutdown hook called
15/11/18 00:24:37 INFO ShutdownHookManager: Deleting directory /private/var/folders/n2/1bnspz7j4q7100jmh610zd200000gn/T/spark-eaa5069c-4f84-4083-bd1a-ede5dc57dc25
[sekikn@mobile bigpetstore-spark]$ hdfs dfs -ls generated_data/transactions
15/11/18 00:24:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
-rw-r--r-- 3 sekikn supergroup 0 2015-11-18 00:24 generated_data/transactions/_SUCCESS
-rw-r--r-- 3 sekikn supergroup 3430043 2015-11-18 00:24 generated_data/transactions/part-00000
-rw-r--r-- 3 sekikn supergroup 3452131 2015-11-18 00:24 generated_data/transactions/part-00001
[sekikn@mobile bigpetstore-spark]$
{code}
> generator.SparkDriver outputs empty data in cluster mode
> --------------------------------------------------------
>
> Key: BIGTOP-2148
> URL: https://issues.apache.org/jira/browse/BIGTOP-2148
> Project: Bigtop
> Issue Type: Bug
> Components: blueprints
> Reporter: Kengo Seki
> Labels: bigpetstore
> Attachments: BIGTOP-2148.00.patch
>
>
> About generator.SparkDriver, bigpetstore-spark/README.md says:
> bq. You will need to change the master if you want to run on a cluster.
> But in reality, if you run it on a cluster, you'll get empty data:
> {code}
> [sekikn@mobile bigpetstore-spark]$ HADOOP_CONF_DIR=/usr/local/Cellar/hadoop/2.7.1/libexec/etc/hadoop spark-submit --master yarn-cluster --class org.apache.bigtop.bigpetstore.spark.generator.SparkDriver build/libs/bigpetstore-spark-1.1.0-SNAPSHOT-all.jar generated_data 10 1000 365.0 345
> (snip)
> 15/11/18 00:12:30 INFO Client: Application report for application_1447772975157_0003 (state: FINISHED)
> 15/11/18 00:12:30 INFO Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: 192.168.0.4
> ApplicationMaster RPC port: 0
> queue: default
> start time: 1447773097856
> final status: SUCCEEDED
> tracking URL: http://mobile.local:8088/proxy/application_1447772975157_0003/
> user: sekikn
> 15/11/18 00:12:30 INFO ShutdownHookManager: Shutdown hook called
> 15/11/18 00:12:30 INFO ShutdownHookManager: Deleting directory /private/var/folders/n2/1bnspz7j4q7100jmh610zd200000gn/T/spark-ccfcde0c-ea95-4361-b2b3-b709a92bee59
> [sekikn@mobile bigpetstore-spark]$ hdfs dfs -ls generated_data/transactions
> 15/11/18 00:13:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Found 3 items
> -rw-r--r-- 3 sekikn supergroup 0 2015-11-18 00:12 generated_data/transactions/_SUCCESS
> -rw-r--r-- 3 sekikn supergroup 0 2015-11-18 00:12 generated_data/transactions/part-00000
> -rw-r--r-- 3 sekikn supergroup 0 2015-11-18 00:12 generated_data/transactions/part-00001
> {code}
> This is because simulationLength is a variable and always -1 in RDD function. It must be a constant or broadcasted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)