[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14534810#comment-14534810 ] 

Steve Loughran commented on SPARK-1537:
---------------------------------------

For people who've not been tracking the WiP

# the timeline API is pretty thoroughly documented with examples; very close to going in
[Latest TimelineServer.md|https://github.com/steveloughran/hadoop-trunk/blob/stevel/YARN-3539-ATS-compatibility/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md]
# the timeline server integration is in sync with trunk, especially the SPARK-4705 changes
# it has lots of tests. This includes: generating events from a spark context and verifying that they are served up by an an-VM timeline server instance, and retrievable by a REST client, bringing up a Spark History server and making GET requests against it to verifying it hooks up to the server, and other cross-system tests. That's about as much as you can do in a standalone unit test suite.
# those tests all run happily on unix and windows, provided you set the {{-Phadoop-2.6 -Pyarn}} flags to request a Hadoop 2.6 profile. 
# and I've tested against hadoop 2.6.0, 2.7.0 &  branch-2; everything compiles and runs

Can I get some reviews?

> Add integration with Yarn's Application Timeline Server
> -------------------------------------------------------
>
>                 Key: SPARK-1537
>                 URL: https://issues.apache.org/jira/browse/SPARK-1537
>             Project: Spark
>          Issue Type: New Feature
>          Components: YARN
>            Reporter: Marcelo Vanzin
>         Attachments: SPARK-1537.txt, spark-1573.patch
>
>
> It would be nice to have Spark integrate with Yarn's Application Timeline Server (see YARN-321, YARN-1530). This would allow users running Spark on Yarn to have a single place to go for all their history needs, and avoid having to manage a separate service (Spark's built-in server).
> At the moment, there's a working version of the ATS in the Hadoop 2.4 branch, although there is still some ongoing work. But the basics are there, and I wouldn't expect them to change (much) at this point.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org