You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tao Li (JIRA)" <ji...@apache.org> on 2015/12/07 13:47:11 UTC
[jira] [Updated] (SPARK-12179) Spark SQL get different result with
the same code
[ https://issues.apache.org/jira/browse/SPARK-12179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tao Li updated SPARK-12179:
---------------------------
Description:
I run the sql in yarn-client mode, but get different result each time.
As you can see the example, I get the different shuffle write with the same shuffle read in two jobs with the same code.
Some of my spark app runs well, but some always met this problem. And I met this problem on spark 1.3, 1.4 and 1.5 version.
Can you give me some suggestions about the possible causes or how do I figure out the problem?
1. First Run
Details for Stage 9 (Attempt 0)
Total Time Across All Tasks: 5.8 min
Shuffle Read: 24.4 MB / 205399
Shuffle Write: 6.8 MB / 54934
2. Second Run
Details for Stage 9 (Attempt 0)
Total Time Across All Tasks: 5.6 min
Shuffle Read: 24.4 MB / 205399
Shuffle Write: 6.8 MB / 54905
was:
I run the sql in yarn-client mode, but get different result each time.
As you can see the example, I get the different shuffle write with the same shuffle read in two jobs.
Some of my spark app runs well, but some always met this problem. And I met this problem on spark 1.3, 1.4 and 1.5 version.
Can you give me some suggestions about the possible causes or how do I figure out the problem?
1. First Run
Details for Stage 9 (Attempt 0)
Total Time Across All Tasks: 5.8 min
Shuffle Read: 24.4 MB / 205399
Shuffle Write: 6.8 MB / 54934
2. Second Run
Details for Stage 9 (Attempt 0)
Total Time Across All Tasks: 5.6 min
Shuffle Read: 24.4 MB / 205399
Shuffle Write: 6.8 MB / 54905
> Spark SQL get different result with the same code
> -------------------------------------------------
>
> Key: SPARK-12179
> URL: https://issues.apache.org/jira/browse/SPARK-12179
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 1.3.0, 1.3.1, 1.3.2, 1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.5.3
> Environment: hadoop version: 2.5.0-cdh5.3.2
> spark version: 1.5.3
> run mode: yarn-client
> Reporter: Tao Li
>
> I run the sql in yarn-client mode, but get different result each time.
> As you can see the example, I get the different shuffle write with the same shuffle read in two jobs with the same code.
> Some of my spark app runs well, but some always met this problem. And I met this problem on spark 1.3, 1.4 and 1.5 version.
> Can you give me some suggestions about the possible causes or how do I figure out the problem?
> 1. First Run
> Details for Stage 9 (Attempt 0)
> Total Time Across All Tasks: 5.8 min
> Shuffle Read: 24.4 MB / 205399
> Shuffle Write: 6.8 MB / 54934
> 2. Second Run
> Details for Stage 9 (Attempt 0)
> Total Time Across All Tasks: 5.6 min
> Shuffle Read: 24.4 MB / 205399
> Shuffle Write: 6.8 MB / 54905
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org