You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "shahid (JIRA)" <ji...@apache.org> on 2018/09/21 10:34:00 UTC
[jira] [Commented] (SPARK-25504) [Spark Job History]If the number
of tasks exceeds spark.ui.retainedTask value, then stage page of UI should
display upto the value set in the retainedTask.
[ https://issues.apache.org/jira/browse/SPARK-25504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623391#comment-16623391 ]
shahid commented on SPARK-25504:
--------------------------------
I am analyzing the issue. Will raise a PR soon.
> [Spark Job History]If the number of tasks exceeds spark.ui.retainedTask value, then stage page of UI should display upto the value set in the retainedTask.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-25504
> URL: https://issues.apache.org/jira/browse/SPARK-25504
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.3.1
> Reporter: ABHISHEK KUMAR GUPTA
> Priority: Major
>
> *Steps*
> 1. spark.ui.retainedTask =60000( spark-default.conf of Job History)
> 2. Run a job for creating 100 task as below from Spark-shell
> val rdd = sc.parallelize(1 to 49000, 49000)
> rdd.count
> 3. Launch Job History Page
> 4. Click on corresponding spark-shell Application ID link in Job History and Launch Job Page for the application
> 5. Click on the Job ID description and Launch Stage Page
> 6. Click on the Description of Stage ID to Launch Stage Page of specific Stage ID
> *Atcual Result*
> Showing Total number of task above Pagination Panel as *Tasks(49000)*
> It is correct as number of task is 49000 created.
> Now Execute as below
> 1. Run a job for creating 100010 task as below from Spark-shell
> val rdd = sc.parallelize(1 to 100100, 100100))
> rdd.count
> 2. Launch Job History Page
> 3. Click on corresponding spark-shell Application ID link in Job History and Launch Job Page for the application
> 4. Click on the Job ID description and Launch Stage Page
> 5. Click on the Description of Stage ID to Launch Stage Page of specific Stage ID
> *Atcual Result*
> Showing Total number of task above Pagination Panel as *Tasks(58100, showing 100100)*
> *Expected:*
> spark.ui.retainedTask =60000
> So it should show as *Tasks( 60000, Showing 100100)* as spark.ui.retainedTask =60000
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org