You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/04/05 03:56:25 UTC
[jira] [Resolved] (SPARK-14240) PySpark Standalone Application
hangs without any Error message
[ https://issues.apache.org/jira/browse/SPARK-14240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-14240.
-------------------------------
Resolution: Not A Problem
Yeah all of this indicates you are not giving your job enough memory. It's not a Spark problem.
> PySpark Standalone Application hangs without any Error message
> --------------------------------------------------------------
>
> Key: SPARK-14240
> URL: https://issues.apache.org/jira/browse/SPARK-14240
> Project: Spark
> Issue Type: Bug
> Components: Deploy, PySpark
> Affects Versions: 1.6.0
> Reporter: Sayak Ghosh
> Attachments: DAG visualisation of pending stages.png, error_terminal.png, event timeline.png
>
>
> I am relatively new to Spark and wrote a simple script using python and spark SQL. My problem is that it is perfectly allright at the starting phase of the execution but gradually it slowed down and at the end of the last phase the whole application hangs
> Here is my code snippet -
> hivectx.registerDataFrameAsTable(aggregatedDataV1,"aggregatedDataV1")
> q1 = "SELECT *, (Total_Sale/Sale_Weeks) as Average_Sale_Per_SaleWeek, (Total_Weeks/Sale_Weeks) as Velocity FROM aggregatedDataV1"
> aggregatedData = hivectx.sql(q1)
> aggregatedData.show(100)
> ========== Terminal Hanging with the following =========
> 16/03/29 09:05:50 INFO TaskSetManager: Finished task 96.0 in stage 416.0 (TID 19992) in 41924 ms on 10.9.0.7 (104/200)
> 16/03/29 09:05:50 INFO TaskSetManager: Finished task 108.0 in stage 416.0 (TID 20004) in 24608 ms on 10.9.0.10 (105/200)
> 16/03/29 09:05:50 INFO TaskSetManager: Finished task 105.0 in stage 416.0 (TID 20001) in 24610 ms on 10.9.0.10 (106/200)
> 16/03/29 09:05:55 INFO TaskSetManager: Starting task 116.0 in stage 416.0 (TID 20012, 10.9.0.10, partition 116,NODE_LOCAL, 2240 bytes)
> 16/03/29 09:06:31 INFO TaskSetManager: Finished task 99.0 in stage 416.0 (TID 19995) in 78435 ms on 10.9.0.7 (110/200)
> 16/03/29 09:06:40 INFO TaskSetManager: Starting task 119.0 in stage 416.0 (TID 20015, 10.9.0.10, partition 119,NODE_LOCAL, 2240 bytes)
> 16/03/29 09:07:12 INFO TaskSetManager: Starting task 122.0 in stage 416.0 (TID 20018, 10.9.0.7, partition 122,NODE_LOCAL, 2240 bytes)
> 16/03/29 09:07:16 INFO TaskSetManager: Starting task 123.0 in stage 416.0 (TID 20019, 10.9.0.7, partition 123,NODE_LOCAL, 2240 bytes)
> 16/03/29 09:07:28 INFO TaskSetManager: Finished task 111.0 in stage 416.0 (TID 20007) in 110198 ms on 10.9.0.7 (114/200)
> 16/03/29 09:07:52 INFO TaskSetManager: Starting task 124.0 in stage 416.0 (TID 20020, 10.9.0.10, partition 124,NODE_LOCAL, 2240 bytes)
> 16/03/29 09:08:08 INFO TaskSetManager: Finished task 110.0 in stage 416.0 (TID 20006) in 150023 ms on 10.9.0.7 (115/200)
> 16/03/29 09:08:12 INFO TaskSetManager: Finished task 113.0 in stage 416.0 (TID 20009) in 154120 ms on 10.9.0.7 (116/200)
> 16/03/29 09:08:16 INFO TaskSetManager: Finished task 116.0 in stage 416.0 (TID 20012) in 145691 ms on 10.9.0.10 (117/200)
> There is no sign of error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org